Start a new topic
Answered

Shub deploy fails, requirements error (I have requirements.txt configured)

I've spent hours trying to find a solution to this.

When trying to do a shub deploy, this happens: 

Entrypoint container is created successfully
>>> Checking python dependencies
awscli 1.10.52 has requirement botocore==1.4.42, but you have botocore 1.10.1.
Warning: Pip checks failed, please fix the conflicts.
{"message": "Dependencies check exit code: 1", "details": "Pip checks failed, please fix the conflicts", "error": "requirements_error"}

{"status": "error", "message": "Requirements error"}
Deploy log location: /var/folders/qc/dzs6cch91lq21cb50q2c7zhm0000gn/T/shub_deploy_e0o02o4v.log
Error: Deploy failed: b'{"status": "error", "message": "Requirements error"}'


 My requirements.txt looks like this:

Distance==0.1.3
nltk==3.2.5
Scrapy==1.5.0
tweepy==3.5.0
scrapinghub==2.0.3
setuptools==36.5.0
gensim==3.4.0
msgpack_python==0.5.6
PyDispatcher==2.0.5


 My scrapinghub.yml looks like this:

project: 285313

requirements:
  file: requirements.txt

stacks:
    default: scrapy:1.1-py3


 It seems that the awscli version is being overrided somehow, as the awscli version that I have installed on my computer is updated. 


Pip freeze, awscli==1.15.1 (on my computer), but when I do a shub deploy, it says that I have awscli==1.10.52


Best Answer

It seems like dependency gensim==3.4.0 enforces downgrading botocore from 1.4.42 to 1.10.1 which breaks awscli requirements. Awscli is part of the 1.1-py3 stack (https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt#L9)


Adding awscli==1.15.1 to your requirements.txt should fix the issue.


Answer

It seems like dependency gensim==3.4.0 enforces downgrading botocore from 1.4.42 to 1.10.1 which breaks awscli requirements. Awscli is part of the 1.1-py3 stack (https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt#L9)


Adding awscli==1.15.1 to your requirements.txt should fix the issue.

This worked, thank you!

I'm glad it worked.

Is there any way just to ignore such warnings?? (like adding a parameter in the .yml file that would take care of that)

No, it's an error that causes conflict when deploying. Otherwise how could the tool determine which version to deploy.

Login to post a comment