When using Zyte Smart Proxy Manager(formerly Crawlera) , it's important to keep in mind the following best practices:


Set download timeout

One of the most common problems our users have is too low download timeout in their web crawlers or scraping application. Handling one request in Smart Proxy Manager can take a long time. This happens due to Smart Proxy Manager internal throttling and it's the way it's supposed to work. Smart Proxy Manager will try to process your request with different slaves and delay time. The recommended timeout for Smart Proxy Manager requests is 600 seconds. If you are using scrapy please check our example configuration.


Adjust concurrency

Adjust concurrency to your plan limit (50 in Basic, 200 in Advanced, etc).


Retry 503 responses

Even though Smart Proxy Manager should protect you against bans, sometimes it runs out of capacity and will return a 503 response. Because of this, we recommend you retry 503 responses up to 5 times. Consider using the x-crawlera-next-request-in to retry more efficiently.


What's the best way to use Smart Proxy Manager with Scrapy?

See Using Smart Proxy Manager with Scrapy.