The recommended way to use Zyte Smart Proxy Manager(formerly Crawlera) with Scrapy is by using the Smart Proxy Manager middleware which can be installed with:

pip install scrapy-zyte-smartproxy

You can enable the middleware by adding the following settings to your Scrapy project:

DOWNLOADER_MIDDLEWARES = {'scrapy_zyte_smartproxy.ZyteSmartProxyMiddleware': 610}
ZYTE_SMARTPROXY_ENABLED = True
ZYTE_SMARTPROXY_APIKEY = '<API key>'

To achieve higher crawl rates when using Smart Proxy Manager with Scrapy, it’s recommended to disable the Auto Throttle addon and increase the maximum number of concurrent requests. You may also want to increase the download timeout. Here is a list of settings that achieve that purpose:

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 32
AUTOTHROTTLE_ENABLED = False
DOWNLOAD_TIMEOUT = 600

To enable Smart Proxy Manager see Getting started with Smart Proxy Manager.


NOTE: You can override Smart Proxy Manager settings from your settings.py file by adding them to Scrapy Cloud project/spider settings (normally, settings entered in Scrapy Cloud UI take precedence over settings entered in the spider, but not in the case of using custom Docker images -- UI settings will be ignored).


For more information, check out our blog post: How to use Smart Proxy Manager with Scrapy.