Scrapy Cloud Advanced Topics

Here you'll find articles on advanced settings and features of Scrapy Cloud.

Downloading and processing images
NOT TO BE CONFUSED WITH THE IMAGES ADDON Scrapy provides reusable item pipelines for downloading images attached to a particular item (for example, when...
Wed, 3 Feb, 2021 at 7:55 AM
Syncing your .scrapy folder to an S3 bucket using DotScrapy Persistence
NOT TO BE CONFUSED WITH THE DOTSCRAPY PERSISTENCE ADDON The purpose of this guide is to keep the content of the .scrapy directory in a persistent store,...
Wed, 3 Feb, 2021 at 7:56 AM
Machine learning with Scrapy and MonkeyLearn
NOT TO BE CONFUSED WITH THE MONKEYLEARN ADDON The purpose of this guide is to bring machine learning technology to the data that you extract through Scr...
Wed, 3 Feb, 2021 at 7:58 AM
Deploy Project and run Spiders with settings of different environments
There are times when we want to test spiders with different configurations/environments but have to deploy spiders to different projects and then test th...
Wed, 3 Feb, 2021 at 7:59 AM
Deploying Custom Docker Image with Selenium on Scrapy Cloud
See shub documentation for the custom Docker image deployment instructions. Chrome Dockerfile Example for Chrome FROM scrapinghub/scrapinghub-st...
Mon, 9 Aug, 2021 at 4:05 PM