Shub deploy fails, requirements error (I have requirements.txt configured)

Posted over 6 years ago by Movere

Post a topic
Answered
M
Movere

I've spent hours trying to find a solution to this.

When trying to do a shub deploy, this happens: 

Entrypoint container is created successfully
>>> Checking python dependencies
awscli 1.10.52 has requirement botocore==1.4.42, but you have botocore 1.10.1.
Warning: Pip checks failed, please fix the conflicts.
{"message": "Dependencies check exit code: 1", "details": "Pip checks failed, please fix the conflicts", "error": "requirements_error"}

{"status": "error", "message": "Requirements error"}
Deploy log location: /var/folders/qc/dzs6cch91lq21cb50q2c7zhm0000gn/T/shub_deploy_e0o02o4v.log
Error: Deploy failed: b'{"status": "error", "message": "Requirements error"}'


 My requirements.txt looks like this:

Distance==0.1.3
nltk==3.2.5
Scrapy==1.5.0
tweepy==3.5.0
scrapinghub==2.0.3
setuptools==36.5.0
gensim==3.4.0
msgpack_python==0.5.6
PyDispatcher==2.0.5


 My scrapinghub.yml looks like this:

project: 285313

requirements:
  file: requirements.txt

stacks:
    default: scrapy:1.1-py3


 It seems that the awscli version is being overrided somehow, as the awscli version that I have installed on my computer is updated. 


Pip freeze, awscli==1.15.1 (on my computer), but when I do a shub deploy, it says that I have awscli==1.10.52

0 Votes

nestor

nestor posted over 6 years ago Admin Best Answer

It seems like dependency gensim==3.4.0 enforces downgrading botocore from 1.4.42 to 1.10.1 which breaks awscli requirements. Awscli is part of the 1.1-py3 stack (https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt#L9)


Adding awscli==1.15.1 to your requirements.txt should fix the issue.

0 Votes


5 Comments

Sorted by
nestor

nestor posted almost 6 years ago Admin

No, it's an error that causes conflict when deploying. Otherwise how could the tool determine which version to deploy.

0 Votes

m

moeen posted almost 6 years ago

Is there any way just to ignore such warnings?? (like adding a parameter in the .yml file that would take care of that)

0 Votes

nestor

nestor posted over 6 years ago Admin

I'm glad it worked.

0 Votes

M

Movere posted over 6 years ago

This worked, thank you!

0 Votes

nestor

nestor posted over 6 years ago Admin Answer

It seems like dependency gensim==3.4.0 enforces downgrading botocore from 1.4.42 to 1.10.1 which breaks awscli requirements. Awscli is part of the 1.1-py3 stack (https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt#L9)


Adding awscli==1.15.1 to your requirements.txt should fix the issue.

0 Votes

Login to post a comment