I use the content from the storage URL (https://storage.scrapinghub.com/activity/nnnnnn/n/nnn?count=1&apikey=[api key]) to know when my task has finished. After starting the crawl, I poll the url every few seconds to check it.
I have just noticed that this isn't being updated in all cases when the job finishes.
It should say e.g.
{"job":"3****4/1/315","event":"job:completed","user":"jobrunner"}
when completed, but it is still reporting e.g.
{"job":"3****1/3/392","event":"job:started","user":"jobrunner"}
tens of minutes after the job has completed.
This all worked fine until 13th Sept, and now doesn't. Any ideas, please?
I have done some more research and re-posted this under a better title.
Could you provide real examples of jobs that are missing from the Activity API or Activity tab in the UI?
Mark Salisbury
I use the content from the storage URL (https://storage.scrapinghub.com/activity/nnnnnn/n/nnn?count=1&apikey=[api key]) to know when my task has finished. After starting the crawl, I poll the url every few seconds to check it.
I have just noticed that this isn't being updated in all cases when the job finishes.
It should say e.g.
{"job":"3****4/1/315","event":"job:completed","user":"jobrunner"}
when completed, but it is still reporting e.g.
{"job":"3****1/3/392","event":"job:started","user":"jobrunner"}
tens of minutes after the job has completed.
This all worked fine until 13th Sept, and now doesn't. Any ideas, please?