Start a new topic

yelp urls was banned requests after some time

Hello,
After near 1 hr of scraping yelp urls f.e this urls format

https://www.yelp.com/biz/the-pink-elephant-alibi-san-francisco-3

future request were blocked/banned.
We made near 80-160 requests per/sec.

screenshot of request/ responce - 503 status

https://prnt.sc/zugo10

Output of crawlera request

curl -vx proxy.crawlera.com:8010 -U ***: https://www.yelp.com/biz/the-pink-elephant-alibi-san-francisco-3

We have plan C 100 (100 concurrent requests). My question is why our server ip were blocked after some time?

if we out of crawlera limitation requests what is happenings of the other requests are crawlera uses proxy on this case?

Thank you!

 

 

 

1 Comment

sorry was 10 requests/sec
on crawlera settings
download_delay = 0.5

concurent_requests = 100

download_timeout = 100

Login to post a comment