Start a new topic

Scrapy complains about missing botocore library when uploading data to S3 bucket

I have already installed botocore library


>> pip freeze

>>boto==2.49.0

boto3==1.18.42

botocore==1.21.42


But when I tried to update data to s3, I still got this error message:


2021-09-16 14:11:32 [scrapy.utils.log] INFO: Scrapy 2.5.0 started

...

...

...

2021-09-16 14:11:32 [scrapy.extensions.feedexport] ERROR: Disabled feed storage scheme: s3. Reason: missing botocore library


My setting files looks like:


AWS_ACCESS_KEY_ID = 'my access key id'

AWS_SECRET_ACCESS_KEY= 'my secrete access key'

FEED_URI='s3://bucke-name/dir-name/filen_ame.json'

ITEM_PIPELINE = {

'scrapy.pipelines.files.S3FilesStore': 1

}


Any idea why the upload failed?

 

 

 

Login to post a comment