So we got the C10 plan and I recently added the Autothrottle to disabled and it went great for one run. But after that we are getting banned on 99% of the requests.
I'm thinking what kind of UA-headers does Crawlera send on the C10 plan? We have a middleware that randomize the UA on the other spiders (not all uses Crawlera) but I don't know if that is passed to Crawlera when it crawls.
Would a idea to try the C50 plan? Or how to get around this? I tried slowing the spider down but it still gets blocked.
Best regards Joacim
Best Answer
n
nestor
said
about 6 years ago
Crawlera rotates the UA from a list of desktop-like UAs by default, so that shouldn't be it. I think you should use your "all" user, I tried several requests with worldwide IPs and had no issues.
Oups, this should have been posted in the Crawlera section. :)
nestor
said
about 6 years ago
Do you need that EU specific geolocation? Does the website not work with IPs from anywhere else?
j
joacimgunnarsson
said
about 6 years ago
Well, from start we got bunch of ips worldwide blocked so we decided to go with only EU-ips from Crawlera but now almost all seems to fail.
So maybe they are looking at the UA?
nestor
said
about 6 years ago
Answer
Crawlera rotates the UA from a list of desktop-like UAs by default, so that shouldn't be it. I think you should use your "all" user, I tried several requests with worldwide IPs and had no issues.
joacimgunnarsson
Hey,
So we got the C10 plan and I recently added the Autothrottle to disabled and it went great for one run. But after that we are getting banned on 99% of the requests.
I'm thinking what kind of UA-headers does Crawlera send on the C10 plan? We have a middleware that randomize the UA on the other spiders (not all uses Crawlera) but I don't know if that is passed to Crawlera when it crawls.
Would a idea to try the C50 plan? Or how to get around this? I tried slowing the spider down but it still gets blocked.
Best regards Joacim
Crawlera rotates the UA from a list of desktop-like UAs by default, so that shouldn't be it. I think you should use your "all" user, I tried several requests with worldwide IPs and had no issues.
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstjoacimgunnarsson
Oups, this should have been posted in the Crawlera section. :)
nestor
Do you need that EU specific geolocation? Does the website not work with IPs from anywhere else?
joacimgunnarsson
Well, from start we got bunch of ips worldwide blocked so we decided to go with only EU-ips from Crawlera but now almost all seems to fail.
So maybe they are looking at the UA?
nestor
Crawlera rotates the UA from a list of desktop-like UAs by default, so that shouldn't be it. I think you should use your "all" user, I tried several requests with worldwide IPs and had no issues.
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 395 topics