Scrapy Cloud Addons
Here you'll find all about the Addons available on Scrapy Cloud.
The Images addon downloads images from extracted image URLs and stores them into an Amazon S3 storage. The addon is enabled by updating the IMAGES_STORE set...
Thu, 11 Feb, 2021 at 10:28 PM
The Auto Throttle addon makes spiders crawl the target sites with more caution, by dynamically adjusting request concurrency and delay according to the site...
Tue, 21 Jun, 2022 at 3:22 PM
⚠ Note that the Monitoring addon is unsupported since 2017. We recommend Spidermon Scrapy extension as an alternative. The Monitoring addon lets you mon...
Wed, 3 Feb, 2021 at 9:21 AM
⚠ The Delta Fetch addon is the Zyte dashboard is deprecated and will be removed soon. You can use the same functionality by using the deltafetch library as ...
Wed, 3 Feb, 2021 at 9:22 AM
If viewing the logs is not enough, the Page Storage Addon could help inspecting the responses Scrapy Cloud is getting from a job's crawl. 1 - Go to ...
Wed, 3 Feb, 2021 at 9:40 AM
The Query Cleaner addon can be used to clean up the request URL GET query parameters at the output of the spider in accordance with the patterns provided by...
Wed, 3 Feb, 2021 at 9:38 AM
This addon keeps the content of the .scrapy directory in a persistent store, which is loaded when the spider starts and saved when the spider finishes. It a...
Wed, 3 Feb, 2021 at 9:38 AM
Sometimes, you need to add certain fields to your scraped data that can be derived from the context. For example, you may need a timestamp for when an item ...
Wed, 3 Feb, 2021 at 9:38 AM
To enable Zyte Smart Proxy Manager(formerly Crawlera) in your Scrapy Cloud project, you can use this addon. To enable it, go to your project, and on the lef...
Tue, 18 Oct, 2022 at 10:22 AM