How to launch a large-scale web scraping project? Find out how LexisNexis did it. Join the webinar on 29th March.Register now

Scrapy Cloud Addons

Here you'll find all about the Addons available on Scrapy Cloud.

Images Storage addon
The Images addon downloads images from extracted image URLs and stores them into an Amazon S3 storage. The addon is enabled by updating the IMAGES_STORE set...
Thu, 11 Feb, 2021 at 10:28 PM
Auto Throttle addon
The Auto Throttle addon makes spiders crawl the target sites with more caution, by dynamically adjusting request concurrency and delay according to the site...
Tue, 21 Jun, 2022 at 3:22 PM
Monitoring addon
⚠ Note that the Monitoring addon is unsupported since 2017. We recommend Spidermon Scrapy extension as an alternative. The Monitoring addon lets you mon...
Wed, 3 Feb, 2021 at 9:21 AM
Delta Fetch addon
⚠ The Delta Fetch addon is the Zyte dashboard is deprecated and will be removed soon. You can use the same functionality by using the deltafetch library as ...
Wed, 3 Feb, 2021 at 9:22 AM
Page Storage addon
If viewing the logs is not enough, the Page Storage Addon could help inspecting the responses Scrapy Cloud is getting from a job's crawl. 1 - Go to...
Wed, 3 Feb, 2021 at 9:40 AM
Query Cleaner addon
The Query Cleaner addon can be used to clean up the request URL GET query parameters at the output of the spider in accordance with the patterns provided by...
Wed, 3 Feb, 2021 at 9:38 AM
DotScrapy Persistence addon
This addon keeps the content of the .scrapy directory in a persistent store, which is loaded when the spider starts and saved when the spider finishes. It a...
Wed, 3 Feb, 2021 at 9:38 AM
Magic Fields addon
Sometimes, you need to add certain fields to your scraped data that can be derived from the context. For example, you may need a timestamp for when an item ...
Wed, 3 Feb, 2021 at 9:38 AM
Zyte Smart Proxy Manager addon
To enable Zyte Smart Proxy Manager(formerly Crawlera) in your Scrapy Cloud project, you can use this addon. To enable it, go to your project, and on the lef...
Tue, 18 Oct, 2022 at 10:22 AM