Start a new topic

yelp urls was banned requests after some time

After near 1 hr of scraping yelp urls f.e this urls format

future request were blocked/banned.
We made near 80-160 requests per/sec.

screenshot of request/ responce - 503 status

Output of crawlera request

curl -vx -U ***:

We have plan C 100 (100 concurrent requests). My question is why our server ip were blocked after some time?

if we out of crawlera limitation requests what is happenings of the other requests are crawlera uses proxy on this case?

Thank you!




1 Comment

sorry was 10 requests/sec
on crawlera settings
download_delay = 0.5

concurent_requests = 100

download_timeout = 100

Login to post a comment