Start a new topic

Spider terminated after one hour

Hello.


as explained here, I have a spider that takes a very long time to process an item (it's a PDF being OCR'd by tesseract).

After one hour of OCR, the spider received a SIGTERM -- which seems consistent with this answer.


Can I avoid this using some kind of signal? Perhaps spider_idle?


Thanks

Login to post a comment