-
Notifications
You must be signed in to change notification settings - Fork 0
simple webcrawler using bs4 and requests.
License
amirongit/webider
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
webider is an absurd attempt to aggregate all the domains that can be found on the internet.
- example proxy config
```
{
"https": "socks5h://host:port",
"http": "http://host:port"
}
```
- how to
```
# after cloning the project, creating a virtual environment and setting up the db uri in config.json
cd webider/src/
pip install -r ../requirements
python main.py
```
- missing dependency for SOCKS?
```
pip install requests[socks]
```
About
simple webcrawler using bs4 and requests.