Start a new topic
Answered

Getting requirements error while doing shub deploy

I am unable to resolve the version conflict between boto3, awscli, s3storage and botocore. I was somehow able to resolve the conflict between these four but i received another error as below


 "watchtower 0.8.0 has requirement boto3<2,>=1.9.253, but you have boto3 1.8.1.

botocore 1.11.1 has requirement urllib3<1.24,>=1.20, but you have urllib3 1.25.10"


The requirements.txt has content as below :

boto3==1.8.1

pandas

unidecode

dateparser

pyes

mmh3

beautifulsoup4

scrapy==1.4.0

ftfy

scrapy-crawlera

elasticsearch==5.1.0

requests-aws4auth

html5lib==1.0b8

six

pytest

watchtower

logzio-python-handler

docker

scrapinghub-entrypoint-scrapy

requests==2.23.0

awscli==1.16.1

botocore==1.11.1

s3transfer==0.1.13


Kindly help to resolve the error


Best Answer

The first one is pretty simple, you are not specifying any watchtower version so it assumes the latest 0.8.0. Watchtower 0.8.0 has a requirement for boto3 to be higher than 1.9.253 and lower than 2. The version you are specifying for boto3 1.8.1 is lower than the one required by watchtower 0.8.0 so it generates a conflict, you need to either put a higher version for boto3 or specify a lower one for watchtower that fills that criteria. 


The second one is similar but a little bit more hidden because you are not specifying urllib3, it is instead part of the 2.0 scrapinghub scrapy stack https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-2.0/requirements.txt#L74. Botocore 1.11.1 has a requirement of urllib3 between 1.20 and 1.24 but the stack predefines 1.25, so you would can specify a higher version for botocore to solve the conflict.


Note that you have a lot of libraries without a specified version, this is not recommended as it causes issues like this.


Answer

The first one is pretty simple, you are not specifying any watchtower version so it assumes the latest 0.8.0. Watchtower 0.8.0 has a requirement for boto3 to be higher than 1.9.253 and lower than 2. The version you are specifying for boto3 1.8.1 is lower than the one required by watchtower 0.8.0 so it generates a conflict, you need to either put a higher version for boto3 or specify a lower one for watchtower that fills that criteria. 


The second one is similar but a little bit more hidden because you are not specifying urllib3, it is instead part of the 2.0 scrapinghub scrapy stack https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-2.0/requirements.txt#L74. Botocore 1.11.1 has a requirement of urllib3 between 1.20 and 1.24 but the stack predefines 1.25, so you would can specify a higher version for botocore to solve the conflict.


Note that you have a lot of libraries without a specified version, this is not recommended as it causes issues like this.

Thanks for the Help !!

Login to post a comment