I am using Scrapy s3 image pipiline to upload images to amazon storage. The log shows no errors whatsoever but only half of the scraped items are uploaded. I have checked scraped items and all of them do have checksum and image url value which means every item has valid image and was processed through pipiline. Where should I look for solution or error? I am under imporession that spider closes before the item pipiline que is finished.
Billy John
I am using Scrapy s3 image pipiline to upload images to amazon storage. The log shows no errors whatsoever but only half of the scraped items are uploaded. I have checked scraped items and all of them do have checksum and image url value which means every item has valid image and was processed through pipiline. Where should I look for solution or error? I am under imporession that spider closes before the item pipiline que is finished.