Zyte Smart Proxy Manager(formerly Crawlera) is a service to download web pages, that supports an HTTP Proxy API.


Standard proxy providers typically provide a pool of IPs running simple HTTP proxies (using Squid or similar software) whereas Smart Proxy Manager downloads web pages, distributing requests among many nodes, keeping track of which nodes are blacklisted (per domain), and throttling them to make sure domains crawled politely, which minimizes the risk of getting your crawler banned.


With proxy providers, you have to implement the throttling and blacklisting logic yourself. With Smart Proxy Manager you only configure your crawler to download pages through Smart Proxy Manager proxy and forget about throttling or implementing anti-ban policies. Smart Proxy Manager enables you to crawl as fast as possible without causing any disruption to the sites.