There's a limit of 1MB for serialised items in Scrapy Cloud (this includes log and request entries). Data pre-processing/normalising is recommended before storing the items (consider writing a pipeline for item sanitising, placed after all other pipelines, or compressing the data with zlib, for instance).
Modified on: Wed, 3 Feb, 2021 at 8:28 AM
Did you find it helpful?Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.