Spider terminated after one hour

Posted over 5 years ago by mredaelli

Post a topic
Un Answered
m
mredaelli

Hello.


as explained here, I have a spider that takes a very long time to process an item (it's a PDF being OCR'd by tesseract).

After one hour of OCR, the spider received a SIGTERM -- which seems consistent with this answer.


Can I avoid this using some kind of signal? Perhaps spider_idle?


Thanks

0 Votes


0 Comments

Login to post a comment