Learn all about the latest trends and best practices in data extraction - Join us at Extract SummitGet tickets
Start a new topic

Limit of downloaded Scrapy files to 100?

Hi - I'm using get("https://app.scrapinghub.com/api/jobs/list.json?" + urlParams, data=crawlRequest, etc) to get a list of jobs to download, however it only returns the most recent 100 jobs.  I can't see any reference to a limit in the documentation, and I have 3700 jobs to download - is there any way to override this limit or do I need to build it into a loop?

1 Comment

How can I download a list of jobs per spider?
Login to post a comment