Zyte Smart Proxy Manager addon

Modified on Tue, 18 Oct, 2022 at 10:22 AM

To enable Zyte Smart Proxy Manager(formerly Crawlera) in your Scrapy Cloud project, you can use this addon. To enable it, go to your project, and on the left panel select Addons Setup (under ADDONS):

Then click on Enable on the Smart Proxy Manager addon:

The addon will appear under your Enabled Addons where you can configure its settings listed below.

Settings

CRAWLERA_URL	proxy URL (default: `http://proxy.crawlera.com:8010`)
CRAWLERA_ENABLED	tick the checkbox to enable Smart Proxy Manager
CRAWLERA_APIKEY	Smart Proxy Manager API key
CRAWLERA_MAXBANS	number of consecutive bans (503s) to ignore before closing the spider (default: `20`)
CRAWLERA_DOWNLOAD_TIMEOUT	timeout for requests (default: `190`)

NOTE #1: To enable Smart Proxy Manager using the scrapy-crawlera middleware, please see: Using Smart Proxy Manager with Scrapy.

NOTE #2: If your project is deployed with a custom Docker image, the ADDONS section will not be displayed. In this case Smart Proxy Manager is enabled in the spider, via the middleware and settings.py file.