So we got the C10 plan and I recently added the Autothrottle to disabled and it went great for one run. But after that we are getting banned on 99% of the requests.
I'm thinking what kind of UA-headers does Crawlera send on the C10 plan? We have a middleware that randomize the UA on the other spiders (not all uses Crawlera) but I don't know if that is passed to Crawlera when it crawls.
Would a idea to try the C50 plan? Or how to get around this? I tried slowing the spider down but it still gets blocked.
Best regards Joacim
0 Votes
nestor posted
almost 7 years ago
AdminBest Answer
Crawlera rotates the UA from a list of desktop-like UAs by default, so that shouldn't be it. I think you should use your "all" user, I tried several requests with worldwide IPs and had no issues.
0 Votes
4 Comments
Sorted by
nestorposted
almost 7 years ago
AdminAnswer
Crawlera rotates the UA from a list of desktop-like UAs by default, so that shouldn't be it. I think you should use your "all" user, I tried several requests with worldwide IPs and had no issues.
0 Votes
j
joacimgunnarssonposted
almost 7 years ago
Well, from start we got bunch of ips worldwide blocked so we decided to go with only EU-ips from Crawlera but now almost all seems to fail.
So maybe they are looking at the UA?
0 Votes
nestorposted
almost 7 years ago
Admin
Do you need that EU specific geolocation? Does the website not work with IPs from anywhere else?
0 Votes
j
joacimgunnarssonposted
almost 7 years ago
Oups, this should have been posted in the Crawlera section. :)
Hey,
So we got the C10 plan and I recently added the Autothrottle to disabled and it went great for one run. But after that we are getting banned on 99% of the requests.
I'm thinking what kind of UA-headers does Crawlera send on the C10 plan? We have a middleware that randomize the UA on the other spiders (not all uses Crawlera) but I don't know if that is passed to Crawlera when it crawls.
Would a idea to try the C50 plan? Or how to get around this? I tried slowing the spider down but it still gets blocked.
Best regards Joacim
0 Votes
nestor posted almost 7 years ago Admin Best Answer
Crawlera rotates the UA from a list of desktop-like UAs by default, so that shouldn't be it. I think you should use your "all" user, I tried several requests with worldwide IPs and had no issues.
0 Votes
4 Comments
nestor posted almost 7 years ago Admin Answer
Crawlera rotates the UA from a list of desktop-like UAs by default, so that shouldn't be it. I think you should use your "all" user, I tried several requests with worldwide IPs and had no issues.
0 Votes
joacimgunnarsson posted almost 7 years ago
Well, from start we got bunch of ips worldwide blocked so we decided to go with only EU-ips from Crawlera but now almost all seems to fail.
So maybe they are looking at the UA?
0 Votes
nestor posted almost 7 years ago Admin
Do you need that EU specific geolocation? Does the website not work with IPs from anywhere else?
0 Votes
joacimgunnarsson posted almost 7 years ago
Oups, this should have been posted in the Crawlera section. :)
0 Votes
Login to post a comment