Can I try Zyte Smart Proxy Manager(formerly Crawlera) before subscribing?
Can I use the Smart Proxy Manager service with my own crawler without using Scrapy, Scrapy Cloud or any other Zyte service?
Definitely. Smart Proxy Manager is a standalone service that can be used with any crawler or HTTP client, independently of the rest of the Zyte platform.
Where is the Smart Proxy Manager API documented?
How do I change my user-agent?
To change your User-Agent you will need to use the X-Crawlera-Profile header with value
pass. This will instruct Smart Proxy Manager to use the User-Agent header you send in the request.
How do I measure Smart Proxy Manager’s speed for a particular domain?
You can use the crawlera-bench tool. Check the GitHub page for more information on how to use it. Note that you would be consuming Smart Proxy Manager traffic when using the
Where can I monitor my Smart Proxy Manager usage?
Go to the Zyte dashboard and select Smart Proxy Manager Overview or a specific account you would like to zoom in. If you click on a user, you will be able to review the number of requests per day/month for that user. Note: Recent Requests section is not populated in real-time. Newly created accounts could be presented with No usage detected so far message, even after the user has started sending requests. Please check the page later to give Scrapy Cloud some time to catch up on the logs.
Why are requests slower through Smart Proxy Manager than other proxies?
If you’re using your own proxies, you may notice a discrepancy in speed between using your own proxies and using them with Smart Proxy Manager. This is because Smart Proxy Manager throttles requests by introducing delays to avoid being banned on the target website.
These delays can differ depending on the target domain, as some popular sites have more rigorous anti-scraping measures than others. Throttling also helps prevent inadvertently bringing down the target website should it lack the resources to handle a large volume of requests.
How does Smart Proxy Manager behave in terms of request headers?
Smart Proxy Manager will make sure it's sending the appropriate headers in a request - Meaning it won't send headers such as x-forwarded-for, or any other proxy-related headers, in order to not appear as a proxy to the target.
You can read more about the X-crawlera headers here.