Zyte Support Center

Is there any way to define a web hook to be called when a spider/job is completed?

If not, could this be added to a feature request list for Scrapy Cloud?

Hi, not sure to understand exactly but could this be helpful?

https://helpdesk.scrapinghub.com/support/solutions/articles/22000200451-getting-notifications-on-certain-events

Best regards,

Pablo

It does not help with automation. The question was about a webhook to be called when the spider is completed. The purpose would be to extract data automatically via the API without the need to constantly call the API and check for changes. 

A spider can take anywhere from 2 minutes to 24 hours. If we poll the API for changes every 2 minutes, it adds up to a lot of calls with an empty result.

I was also trying to figure out if this was possible, and I found one way to accomplish webhook functionality is to use the closed() method of a spider:

https://doc.scrapy.org/en/latest/topics/spiders.html?highlight=closed#scrapy.spiders.Spider.closed

This closed() method will be called after the spider is done, so you can make a request to another url here. And since you can pass spiders arguments, you can provide a custom webhook url to notify you when the spider is completed. Hope that helps.

I was relying on the Spider.closed callback as mentioned above but, the problem is, the job isn't finished when this is called because the spider itself hasn't exited yet. It's kind of strange for a platform like this not to have any webhook support.