videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic
Answered

What the settings for scrapy and crawlera?

What the settings I need add to scrapy project if I want to use crawlera? for example c 50 plan? If I want quick speed scrape webpages?


Best Answer

Start with Crawlera best practices:  https://support.scrapinghub.com/support/solutions/articles/22000188397-crawlera-best-practices 
And if you have any issues, you can open a ticket from the dashboard: Help > Contact support.


Answer

Start with Crawlera best practices:  https://support.scrapinghub.com/support/solutions/articles/22000188397-crawlera-best-practices 
And if you have any issues, you can open a ticket from the dashboard: Help > Contact support.

You can increase the speed my increasing concurrency. You can set concurrency using CONCURRNT_REQUESTS and disabling Autothrottle as given in https://support.scrapinghub.com/solution/articles/22000188399-using-crawlera-with-scrapy.

Login to post a comment