Start a new topic
Answered

Shub deploy fails, requirements error, requirements.txt is configured

I've been trying for hours to deploy this spider, when trying to do a shub deploy in verbose mode, this is the error message I receive: 


$ shub deploy 344151 --verbose
Packing version 835d360-master
Deploying to Scrapy Cloud project "344151"
Login succeeded
Building an image:
Step 1/12 : FROM scrapinghub/scrapinghub-stack-scrapy:1.1-py3
# Executing 2 build triggers...

Step 1/1 : ENV PIP_TRUSTED_HOST $PIP_TRUSTED_HOST PIP_INDEX_URL $PIP_INDEX_URL
 ---> Using cache
Step 1/1 : RUN test -n $APT_PROXY && echo 'Acquire::http::Proxy \"$APT_PROXY\";'     >/etc/apt/apt.conf.d/proxy
 ---> Using cache
 ---> 5007fbdbcfa4
Step 2/12 : ENV PYTHONUSERBASE /app/python
 ---> Using cache
 ---> 6780a1586fdc
Step 3/12 : ADD eggbased-entrypoint /usr/local/sbin/
 ---> Using cache
 ---> d58959161694
Step 4/12 : ADD run-pipcheck /usr/local/bin/
 ---> Using cache
 ---> 1676330f0f58
Step 5/12 : RUN chmod +x /usr/local/bin/run-pipcheck
 ---> Using cache
 ---> cbe9621799ba
Step 6/12 : RUN chmod +x /usr/local/sbin/eggbased-entrypoint &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/start-crawl &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/scrapy-list &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/shub-image-info &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/run-pipcheck
 ---> Using cache
 ---> 661148411a2d
Step 7/12 : ADD requirements.txt /app/requirements.txt
 ---> f586387e3b29
Removing intermediate container 10a6a9e794cb
Step 8/12 : RUN mkdir /app/python && chown nobody:nogroup /app/python
 ---> Running in 47c96d199ba4
 ---> e15cfe130b2f
Removing intermediate container 47c96d199ba4
Step 9/12 : RUN sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --user --no-cache-dir -r /app/requirements.txt
 ---> Running in 7bab963d71d3
Requirement already satisfied (use --upgrade to upgrade): scrapy==1.1.3 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 1))
Collecting boto==2.47.0 (from -r /app/requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/d7/ce/ced61cdf36d09038fc4c88bac1182845489e51a950148f5c1d1a4804cc28/boto-2.47.0-py2.py3-none-any.whl (1.4MB)
Collecting extruct (from -r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/da/73/2ce798fd214c988f06bc78256d06647596cca5632bdde4456a29f3a3fb48/extruct-0.7.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): w3lib==1.14.3 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): jinja2==2.8 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 5))
Collecting awscli==1.15.1 (from -r /app/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/0a/c2/93cc87bb61c62646ec929d1ebb466b5fa47fe8e98ec6971f5d4dbefc6c7a/awscli-1.15.1-py2.py3-none-any.whl (1.2MB)
Requirement already satisfied (use --upgrade to upgrade): idna==2.1 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): PyDispatcher>=2.0.5 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): service-identity in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): parsel>=0.9.3 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Collecting mf2py (from extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/fe/09/116b5b42f5c8cbb1cc43408be9df12fc2bf9fc408dfc9b20311ef4ffbfec/mf2py-1.1.2.tar.gz
Collecting rdflib-jsonld (from extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/ba/48/edaad22fc9de34500699f0c7fc9124385dd425fd18857244f126a6f7df66/rdflib-jsonld-0.4.0.tar.gz (53kB)
Collecting rdflib (from extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/3c/fe/630bacb652680f6d481b9febbb3e2c3869194a1a5fc3401a4a41195a2f8f/rdflib-4.2.2-py3-none-any.whl (344kB)
Requirement already satisfied (use --upgrade to upgrade): MarkupSafe in /usr/local/lib/python3.5/site-packages (from jinja2==2.8->-r /app/requirements.txt (line 5))
Collecting s3transfer<0.2.0,>=0.1.12 (from awscli==1.15.1->-r /app/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/d7/14/2a0004d487464d120c9fb85313a75cd3d71a7506955be458eebfe19a6b1d/s3transfer-0.1.13-py2.py3-none-any.whl (59kB)
Collecting botocore==1.10.1 (from awscli==1.15.1->-r /app/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/ae/e9/304712de6242f0f883dc358adbeee0e05b51d2f538d4bff645dbe11b616e/botocore-1.10.1-py2.py3-none-any.whl (4.2MB)
Requirement already satisfied (use --upgrade to upgrade): rsa<=3.5.0,>=3.1.2 in /usr/local/lib/python3.5/site-packages (from awscli==1.15.1->-r /app/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): colorama<=0.3.7,>=0.2.5 in /usr/local/lib/python3.5/site-packages (from awscli==1.15.1->-r /app/requirements.txt (line 6))
Collecting PyYAML<=3.12,>=3.10 (from awscli==1.15.1->-r /app/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz (253kB)
Requirement already satisfied (use --upgrade to upgrade): docutils>=0.10 in /usr/local/lib/python3.5/site-packages (from awscli==1.15.1->-r /app/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /usr/local/lib/python3.5/site-packages (from service-identity->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): attrs in /usr/local/lib/python3.5/site-packages (from service-identity->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /usr/local/lib/python3.5/site-packages (from service-identity->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=4.0.2 in /usr/local/lib/python3.5/site-packages (from Twisted>=10.0.0->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): cryptography>=1.3 in /usr/local/lib/python3.5/site-packages (from pyOpenSSL->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Collecting html5lib>=1.0.1 (from mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl (117kB)
Collecting requests>=2.18.4 (from mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl (91kB)
Collecting BeautifulSoup4>=4.6.0 (from mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/21/0a/47fdf541c97fd9b6a610cb5fd518175308a7cc60569962e776ac52420387/beautifulsoup4-4.6.3-py3-none-any.whl (90kB)
Collecting isodate (from rdflib->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl (45kB)
Requirement already satisfied (use --upgrade to upgrade): pyparsing in /usr/local/lib/python3.5/site-packages (from rdflib->extruct->-r /app/requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): python-dateutil<2.7.0,>=2.1 in /usr/local/lib/python3.5/site-packages (from botocore==1.10.1->awscli==1.15.1->-r /app/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.5/site-packages (from botocore==1.10.1->awscli==1.15.1->-r /app/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python3.5/site-packages (from zope.interface>=4.0.2->Twisted>=10.0.0->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.4.1 in /usr/local/lib/python3.5/site-packages (from cryptography>=1.3->pyOpenSSL->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Collecting webencodings (from html5lib>=1.0.1->mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.18.4->mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting urllib3<1.24,>=1.21.1 (from requests>=2.18.4->mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB)
Collecting certifi>=2017.4.17 (from requests>=2.18.4->mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl (147kB)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/local/lib/python3.5/site-packages (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Installing collected packages: boto, webencodings, html5lib, chardet, urllib3, certifi, requests, BeautifulSoup4, mf2py, isodate, rdflib, rdflib-jsonld, extruct, botocore, s3transfer, PyYAML, awscli
  Running setup.py install for mf2py: started
    Running setup.py install for mf2py: finished with status 'done'
  Running setup.py install for rdflib-jsonld: started
    Running setup.py install for rdflib-jsonld: finished with status 'done'
  Running setup.py install for PyYAML: started
    Running setup.py install for PyYAML: finished with status 'done'
Successfully installed BeautifulSoup4 PyYAML awscli-1.10.52 boto-2.42.0 botocore-1.4.42 certifi chardet extruct html5lib isodate mf2py rdflib rdflib-jsonld requests-2.10.0 s3transfer-0.1.1 urllib3 webencodings
You are using pip version 8.1.2, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

 ---> 6228de97ecda
Removing intermediate container 7bab963d71d3
Step 10/12 : COPY *.egg /app/
 ---> 2801677070a2
Removing intermediate container 36861c9d6e30
Step 11/12 : RUN if [ -d "/app/addons_eggs" ]; then rm -f /app/*.dash-addon.egg; fi
 ---> Running in fccb3f29874b
 ---> 1ec4b4e132b6
Removing intermediate container fccb3f29874b
Step 12/12 : ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 ---> Running in 43ccb7fd3f30
 ---> 583810d67794
Removing intermediate container 43ccb7fd3f30
Successfully built 583810d67794
Step 1/3 : FROM alpine:3.5
 ---> 6c6084ed97e5
Step 2/3 : ADD kumo-entrypoint /kumo-entrypoint
 ---> Using cache
 ---> 4e595e8e525b
Step 3/3 : RUN chmod +x /kumo-entrypoint
 ---> Using cache
 ---> 6a61902913ac
Successfully built 6a61902913ac
Entrypoint container is created successfully
>>> Checking python dependencies
Collecting pip==9.0.3
  Downloading https://files.pythonhosted.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
Successfully installed pip-9.0.3
requests 2.19.1 has requirement idna<2.8,>=2.5, but you have idna 2.1.
Warning: Pip checks failed, please fix the conflicts.
{"message": "Dependencies check exit code: 1", "details": "Pip checks failed, please fix the conflicts", "error": "requirements_error"}

{"status": "error", "message": "Requirements error"}
Deploy log location: /var/folders/5p/ys4j0bl16lqgrx5hxyplmykc0000gn/T/shub_deploy_9x0hmgdt.log
Error: Deploy failed: b'{"status": "error", "message": "Requirements error"}'

 My requirements.txt


scrapy==1.1.3
boto==2.47.0
extruct
w3lib==1.14.3             # via parsel, scrapy
jinja2==2.8

 Please can you help.






 

 

 

 

 

 

 

 


Best Answer

I would start by setting a version for your requirements, it's generally bad practice to not specify the version.


The error is specified in the last few lines of the log:


requests 2.19.1 has requirement idna<2.8,>=2.5, but you have idna 2.1.


The stack you're using, scrapy1.1-py3, has idna 2.1 as a requirement: https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt and it conflicts with the requests version being installed. So try setting idna= some version between 2.8 and 2.5


Or you can also use a newer py3 stack like scrapy1.5-py3.



Answer

I would start by setting a version for your requirements, it's generally bad practice to not specify the version.


The error is specified in the last few lines of the log:


requests 2.19.1 has requirement idna<2.8,>=2.5, but you have idna 2.1.


The stack you're using, scrapy1.1-py3, has idna 2.1 as a requirement: https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt and it conflicts with the requests version being installed. So try setting idna= some version between 2.8 and 2.5


Or you can also use a newer py3 stack like scrapy1.5-py3.


Thanks Nestor,


I updated to scrapy1.5.1 and requirements.txt now reads:


scrapy==1.5.1
boto==2.42.0
extruct==0.1.0
w3lib>=1.17.0     # via parsel, scrapy
jinja2==2.8
idna==2.6        # via cryptography


Deploys OK. Thank you

I'm glad it worked :)

Login to post a comment