To enable Zyte Smart Proxy Manager(formerly Crawlera) in your Scrapy Cloud project, you can use this addon. To enable it, go to your project, and on the left panel select Addons Setup (under ADDONS):
Then click on Enable on the Smart Proxy Manager addon:
The addon will appear under your Enabled Addons where you can configure its settings listed below.
Settings
| CRAWLERA_URL | proxy URL (default: http://proxy.crawlera.com:8010) |
| CRAWLERA_ENABLED | tick the checkbox to enable Smart Proxy Manager |
| CRAWLERA_APIKEY | Smart Proxy Manager API key |
| CRAWLERA_MAXBANS | number of consecutive bans (503s) to ignore before closing the spider (default: 20) |
| CRAWLERA_DOWNLOAD_TIMEOUT | timeout for requests (default: 190) |
NOTE #1: To enable Smart Proxy Manager using the scrapy-crawlera middleware, please see: Using Smart Proxy Manager with Scrapy.
NOTE #2: If your project is deployed with a custom Docker image, the ADDONS section will not be displayed. In this case Smart Proxy Manager is enabled in the spider, via the middleware and settings.py file.