No recent searches
Popular Articles
Sorry! nothing found for
Posted over 3 years ago by Roman Grabar
Hello, After near 1 hr of scraping yelp urls f.e this urls format
https://www.yelp.com/biz/the-pink-elephant-alibi-san-francisco-3
future request were blocked/banned. We made near 80-160 requests per/sec.
screenshot of request/ responce - 503 status
https://prnt.sc/zugo10
Output of crawlera request
curl -vx proxy.crawlera.com:8010 -U ***: https://www.yelp.com/biz/the-pink-elephant-alibi-san-francisco-3
We have plan C 100 (100 concurrent requests). My question is why our server ip were blocked after some time?
if we out of crawlera limitation requests what is happenings of the other requests are crawlera uses proxy on this case?
Thank you!
0 Votes
1 Comments
Roman Grabar posted over 3 years ago
sorry was 10 requests/secon crawlera settingsdownload_delay = 0.5
concurent_requests = 100
download_timeout = 100
Login to post a comment
People who like this
This post will be deleted permanently. Are you sure?
Hello,
After near 1 hr of scraping yelp urls f.e this urls format
https://www.yelp.com/biz/the-pink-elephant-alibi-san-francisco-3
future request were blocked/banned.
We made near 80-160 requests per/sec.
screenshot of request/ responce - 503 status
https://prnt.sc/zugo10
Output of crawlera request
curl -vx proxy.crawlera.com:8010 -U ***: https://www.yelp.com/biz/the-pink-elephant-alibi-san-francisco-3
We have plan C 100 (100 concurrent requests). My question is why our server ip were blocked after some time?
if we out of crawlera limitation requests what is happenings of the other requests are crawlera uses proxy on this case?
Thank you!
0 Votes
1 Comments
Roman Grabar posted over 3 years ago
sorry was 10 requests/sec
on crawlera settings
download_delay = 0.5
concurent_requests = 100
download_timeout = 100
0 Votes
Login to post a comment