There's a limit of 1MB for serialised items in Scrapy Cloud (this includes log and request entries). Data pre-processing/normalising is recommended before storing the items (consider writing a pipeline for item sanitising, placed after all other pipelines, or compressing the data with zlib, for instance).
Why do I get "Rejected message because it was too big" error?
Modified on: Wed, 3 Feb, 2021 at 8:28 AM
Did you find it helpful? Yes No
Send feedbackSorry we couldn't be helpful. Help us improve this article with your feedback.