Start a new topic
Answered

Almost all requests are getting blocket

Hey,

So we got the C10 plan and I recently added the Autothrottle to disabled and it went great for one run. But after that we are getting banned on 99% of the requests.

I'm thinking what kind of UA-headers does Crawlera send on the C10 plan? We have a middleware that randomize the UA on the other spiders (not all uses Crawlera) but I don't know if that is passed to Crawlera when it crawls. 

Would a idea to try the C50 plan? Or how to get around this? I tried slowing the spider down but it still gets blocked.

Best regards Joacim


Best Answer

Crawlera rotates the UA from a list of desktop-like UAs by default, so that shouldn't be it. I think you should use your "all" user, I tried several requests with worldwide IPs and had no issues.


Oups, this should have been posted in the Crawlera section. :) 

Do you need that EU specific geolocation? Does the website not work with IPs from anywhere else?

Well, from start we got bunch of ips worldwide blocked so we decided to go with only EU-ips from Crawlera but now almost all seems to fail. 

So maybe they are looking at the UA? 

Answer

Crawlera rotates the UA from a list of desktop-like UAs by default, so that shouldn't be it. I think you should use your "all" user, I tried several requests with worldwide IPs and had no issues.

Login to post a comment