videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic
Answered

Boto3

Hi I try to set up my script wiht boto3 to scrapinghub cloud, in deploy it does not work, I encounter some version error for pip install.


How can I solve it?


Best Answer

Oh not really, but I figured things out here, I put

botocore==1.5.8

boto3==1.4.4

and it works fine now. 


Thanks a lot for your help!


Can you show the requirements file? Did you hardcode the versions in the requirements.txt?

No I just put boto3 in it, how could I know what version I should use for boto3?

image


You can add versions to the `requirements.txt` with boto3==10.0.0, does that help?

it says  Could not find a version that satisfies the requirement boto3==10.0.0

Sorry that was an example. From the screen shot you sent you want


awscli==1.11.45

botocore==1.5.8

s3transfer==0.1.9


What is the code you use to upload to S3?

upload to S3? sorry you mean shub deploy?

I assumed you use boto and aws to upload your scraped items to S3. 

Answer

Oh not really, but I figured things out here, I put

botocore==1.5.8

boto3==1.4.4

and it works fine now. 


Thanks a lot for your help!


1 person likes this
Login to post a comment