Scrapy Cloud FAQ

Which services are available on Scrapy Cloud?
Note: Portia is no longer available for new users. It has been disabled for all the new organisations from August 20, 2018 onward. Here is a diagram tha...
Wed, 3 Feb, 2021 at 8:21 AM
What is a Scrapy Cloud Unit?
Scrapy Cloud jobs run in containers. These containers can be of different sizes defined by Scrapy Cloud units. A Scrapy Cloud provides: 1 GB of RAM 2.5G...
Thu, 27 Jan, 2022 at 5:40 PM
Why do I see only one Scrapy Cloud unit in Dashboard after purchasing an extra unit?
When you subscribe to Zyte, 1 Scrapy Cloud unit is given for free. This free unit has features such as data retention for 7 days and a run time limit for jo...
Wed, 3 Feb, 2021 at 8:24 AM
Can I use an HTTP cache on Scrapy Cloud?
Yes, you can. To do that, you have to enable Scrapy's HTTP cache extension by setting HTTPCACHE_ENABLED to True in your project settings. The...
Wed, 3 Feb, 2021 at 8:24 AM
What are the IP addresses used by Scrapy Cloud?
Scenario: You have e.g. a MongoDB server that you would like your spiders to write to, and you would like to open access to that server only from Scrapy Clo...
Wed, 3 Feb, 2021 at 8:25 AM
Can I run Scrapy Cloud on my environment?
Scrapy Cloud is only offered as a hosted service. If you're interested in hosting your own version of Scrapy Cloud, get in touch and let us know.
Wed, 3 Feb, 2021 at 8:25 AM
What are the differences between running a spider locally and on Scrapy Cloud?
Here are a few things that work differently on Scrapy Cloud, compared to a default Scrapy configuration:  AutoThrottle extension is enabled, to crawl webs...
Wed, 18 Oct, 2023 at 12:46 PM
Where can I find my activation (or re-activation) link?
A common issue is not to receive your activation link when you subscribe to a new account in Zyte or when you forgot your password and trying to send  re-ac...
Wed, 3 Feb, 2021 at 8:26 AM
How can I migrate my Python eggs to Scrapy Cloud 2.0?
Zyte will support eggs within the Zyte Dashboard through the end of 2016. After that, project dependencies outside of the Scrapy Cloud stack must be managed...
Wed, 3 Feb, 2021 at 8:27 AM
Scrapy Cloud vs Scrapyd (using Heroku)
Here at Zyte we are big fans of Heroku. When people ask what Scrapy Cloud is about we sometimes tell people that "it's like Heroku, but for web cra...
Wed, 3 Feb, 2021 at 8:28 AM