Limit of downloaded Scrapy files to 100?

Posted about 4 years ago by Phil Clark

Post a topic
Un Answered
P
Phil Clark

Hi - I'm using get("https://app.scrapinghub.com/api/jobs/list.json?" + urlParams, data=crawlRequest, etc) to get a list of jobs to download, however it only returns the most recent 100 jobs.  I can't see any reference to a limit in the documentation, and I have 3700 jobs to download - is there any way to override this limit or do I need to build it into a loop?

0 Votes


1 Comments

p

prismawebscraping posted over 2 years ago

How can I download a list of jobs per spider?

0 Votes

Login to post a comment