Start a new topic

Only half of the images uploaded to s3. No error

I am using Scrapy s3 image pipiline to upload images to amazon storage. The log shows no errors whatsoever but only half of the scraped items are uploaded. I have checked scraped items and all of them do have checksum and image url value which means every item has valid image and was processed through pipiline. Where should I look for solution or error? I am under imporession that spider closes before the item pipiline que is finished.

(33.2 KB)
Login to post a comment