Monitoring addon

⚠ Note that the Monitoring addon is unsupported since 2017. We recommend Spidermon Scrapy extension as an alternative.

The Monitoring addon lets you monitor your spiders, generate reports and send notifications. It works by allowing you to set one or more actions to execute on a certain job event.

How to use it

Go to your project addons page and enable the Monitoring addon. If you want the monitor to send notifications via Slack, you have to create a bot user under your Slack's organization (or reuse one) and configure it in the addon settings page:

Next, you have to go to the Monitoring page for your project:

In this page, you can create Monitors or configure Actions.

Monitors

A Monitor consists of one or more tests that check one or more conditions. For example, a monitor could check if the number of scraped items is above a certain threshold and add a fail message into a report if it's not true after the spider finishes.

To setup a monitor, you have to click the "Add monitor" button in the Monitoring landing page and after that, you have to add the tests to the monitor:

Then, in the next screen you'll be able to create a test. For each test you have to provide an assertion and a fail message. The example below checks if a spider finished successfully:

Then, when you run your spider, this test will be executed and if the assertion fails the fail message will be included in a report that will be sent to you if you configure an action for that.

Actions

You can also configure an Action to be executed when a certain event is triggered in your jobs. The following events are available:

• Spider started: Run actions when a spider starts
• Spider finished: Run actions when a spider finishes
• Monitors passed: Run actions when all monitors pass
• Monitors failed: Run actions when one or more monitors fail.

For example, you can setup an action to send you an email or Slack message with a report once the spider finishes.

This way, you'll get a message like this in your inbox once the spider finishes:

Enabling the Monitors/Actions

After you create your monitors and actions, you have to enable them for each of your spiders, otherwise no notifications will be sent. You can do so in each of the spider's dashboards, as shown below.