I encountered this error when using Scrapy Cloud to run my job:
Rejected message because it was too big: ITM {...
I am sure that the request is fired successfully. This error should be caused by large response, which is about 2M. But this is the size I need to store.
I followed the advices in the following link, i.e. enabling Page Storage addon, and configured a Amazon S3 storage using Feed Export:
However, the same error is throw. May i know how to solve this issue?
1 Votes
1 Comments
R
Robb Shecterposted
over 4 years ago
I have the same issue. My scraper produces just one result item, which is a 100MB JSON tree representing a website's data. This works fine locally for me.
What limits are enforced? Is there any way to use Scrapy Cloud with an item like this?
Dear Support Team,
I encountered this error when using Scrapy Cloud to run my job:
Rejected message because it was too big: ITM {...
I am sure that the request is fired successfully. This error should be caused by large response, which is about 2M. But this is the size I need to store.
I followed the advices in the following link, i.e. enabling Page Storage addon, and configured a Amazon S3 storage using Feed Export:
https://support.scrapinghub.com/support/discussions/topics/22000009523
However, the same error is throw. May i know how to solve this issue?
1 Votes
1 Comments
Robb Shecter posted over 4 years ago
I have the same issue. My scraper produces just one result item, which is a 100MB JSON tree representing a website's data. This works fine locally for me.
What limits are enforced? Is there any way to use Scrapy Cloud with an item like this?
0 Votes
Login to post a comment