Hi - I'm using get("https://app.scrapinghub.com/api/jobs/list.json?" + urlParams, data=crawlRequest, etc) to get a list of jobs to download, however it only returns the most recent 100 jobs. I can't see any reference to a limit in the documentation, and I have 3700 jobs to download - is there any way to override this limit or do I need to build it into a loop?
Phil Clark
Hi - I'm using get("https://app.scrapinghub.com/api/jobs/list.json?" + urlParams, data=crawlRequest, etc) to get a list of jobs to download, however it only returns the most recent 100 jobs. I can't see any reference to a limit in the documentation, and I have 3700 jobs to download - is there any way to override this limit or do I need to build it into a loop?