Why do I get "Rejected message because it was too big" error?

Modified on Wed, 3 Feb, 2021 at 8:28 AM

There's a limit of 1MB for serialised items in Scrapy Cloud (this includes log and request entries). Data pre-processing/normalising is recommended before storing the items (consider writing a pipeline for item sanitising, placed after all other pipelines, or compressing the data with zlib, for instance).


Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select at least one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article