yelp urls was banned requests after some time

Posted over 3 years ago by Roman Grabar

Post a topic
Un Answered
R
Roman Grabar

Hello,
After near 1 hr of scraping yelp urls f.e this urls format

https://www.yelp.com/biz/the-pink-elephant-alibi-san-francisco-3

future request were blocked/banned.
We made near 80-160 requests per/sec.

screenshot of request/ responce - 503 status

https://prnt.sc/zugo10

Output of crawlera request

curl -vx proxy.crawlera.com:8010 -U ***: https://www.yelp.com/biz/the-pink-elephant-alibi-san-francisco-3

We have plan C 100 (100 concurrent requests). My question is why our server ip were blocked after some time?

if we out of crawlera limitation requests what is happenings of the other requests are crawlera uses proxy on this case?

Thank you!

 

 

 

0 Votes


1 Comments

R

Roman Grabar posted over 3 years ago

sorry was 10 requests/sec
on crawlera settings
download_delay = 0.5

concurent_requests = 100

download_timeout = 100

0 Votes

Login to post a comment