I am using Scrapy s3 image pipiline to upload images to amazon storage. The log shows no errors whatsoever but only half of the scraped items are uploaded. I have checked scraped items and all of them do have checksum and image url value which means every item has valid image and was processed through pipiline. Where should I look for solution or error? I am under imporession that spider closes before the item pipiline que is finished.
I am using Scrapy s3 image pipiline to upload images to amazon storage. The log shows no errors whatsoever but only half of the scraped items are uploaded. I have checked scraped items and all of them do have checksum and image url value which means every item has valid image and was processed through pipiline. Where should I look for solution or error? I am under imporession that spider closes before the item pipiline que is finished.
Attachments (1)
1.png
33.2 KB
0 Votes
0 Comments
Login to post a comment