After near 1 hr of scraping yelp urls f.e this urls format
future request were blocked/banned.
We made near 80-160 requests per/sec.
screenshot of request/ responce - 503 status
Output of crawlera request
curl -vx proxy.crawlera.com:8010 -U ***: https://www.yelp.com/biz/the-pink-elephant-alibi-san-francisco-3
We have plan C 100 (100 concurrent requests). My question is why our server ip were blocked after some time?
if we out of crawlera limitation requests what is happenings of the other requests are crawlera uses proxy on this case?
sorry was 10 requests/sec
on crawlera settings
download_delay = 0.5
concurent_requests = 100
download_timeout = 100