What the settings for scrapy and crawlera?

a

started a topic almost 4 years ago

What the settings I need add to scrapy project if I want to use crawlera? for example c 50 plan? If I want quick speed scrape webpages?

Best Answer

n

nestor said almost 4 years ago

Start with Crawlera best practices: https://support.scrapinghub.com/support/solutions/articles/22000188397-crawlera-best-practices
And if you have any issues, you can open a ticket from the dashboard: Help > Contact support.

said almost 4 years ago

Answer

Start with Crawlera best practices: https://support.scrapinghub.com/support/solutions/articles/22000188397-crawlera-best-practices
And if you have any issues, you can open a ticket from the dashboard: Help > Contact support.

said almost 4 years ago

You can increase the speed my increasing concurrency. You can set concurrency using CONCURRNT_REQUESTS and disabling Autothrottle as given in https://support.scrapinghub.com/solution/articles/22000188399-using-crawlera-with-scrapy.

Zyte Support Center