I run the crawler, about 16 hours a night, making 478 requests, very slow.I wonder why?When my crawler wasn't running on a messy cloud, it was ok, but Google banned it, so I used cralera, but the efficiency was worrying.
See solution 1) from: https://support.scrapinghub.com/a/forums/topics/22000010595
lisisi2018
I run the crawler, about 16 hours a night, making 478 requests, very slow.I wonder why?When my crawler wasn't running on a messy cloud, it was ok, but Google banned it, so I used cralera, but the efficiency was worrying.
See solution 1) from: https://support.scrapinghub.com/a/forums/topics/22000010595
nestor
See solution 1) from: https://support.scrapinghub.com/a/forums/topics/22000010595
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 365 topics