Only half of the images uploaded to s3. No error

Posted over 6 years ago by Artem M

Post a topic
Un Answered
A
Artem M

I am using Scrapy s3 image pipiline to upload images to amazon storage. The log shows no errors whatsoever but only half of the scraped items are uploaded. I have checked scraped items and all of them do have checksum and image url value which means every item has valid image and was processed through pipiline. Where should I look for solution or error? I am under imporession that spider closes before the item pipiline que is finished.


Attachments (1)

0 Votes


0 Comments

Login to post a comment