On the official Scrapy docs there is a section for "pausing and resuming crawls". It states that through the JOBDIR setting it is possible to resume the state of a job. I've also found a Zyte's FAQ that says "JOBDIRis set, causing the scheduler (requests queue) to be persisted on disk, and save memory".
Is there a way to override JOBDIR in order to resume a stopped job from where it terminated its execution?
On the official Scrapy docs there is a section for "pausing and resuming crawls". It states that through the JOBDIR setting it is possible to resume the state of a job. I've also found a Zyte's FAQ that says "JOBDIR is set, causing the scheduler (requests queue) to be persisted on disk, and save memory".
Is there a way to override JOBDIR in order to resume a stopped job from where it terminated its execution?
0 Votes
0 Comments
Login to post a comment