videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic

Job is running slowly.

I have i c50 crawlera plan that runs at 50 threads and quite basic spider.

But it runs impossibly slow. Like 25 requests per minute.

The only thing is that i load 250k start urls from google sheet. Can this influence the speed? Autothrotle is disabled of course.

Login to post a comment