Recent Searches

No recent searches

    Popular Articles

      Articles
      View all
        Topics
        View all
          Tickets
          View all
            no results

            Sorry! nothing found for

            Shub deploy fails, requirements error, requirements.txt is configured

            Posted over 6 years ago by Jonathan Bowker

            Post a topic
            Answered
            J
            Jonathan Bowker
            Post a topic

            I've been trying for hours to deploy this spider, when trying to do a shub deploy in verbose mode, this is the error message I receive:  


            $ shub deploy 344151 --verbose
Packing version 835d360-master
Deploying to Scrapy Cloud project "344151"
Login succeeded
Building an image:
Step 1/12 : FROM scrapinghub/scrapinghub-stack-scrapy:1.1-py3
# Executing 2 build triggers...

Step 1/1 : ENV PIP_TRUSTED_HOST $PIP_TRUSTED_HOST PIP_INDEX_URL $PIP_INDEX_URL
 ---> Using cache
Step 1/1 : RUN test -n $APT_PROXY && echo 'Acquire::http::Proxy \"$APT_PROXY\";'     >/etc/apt/apt.conf.d/proxy
 ---> Using cache
 ---> 5007fbdbcfa4
Step 2/12 : ENV PYTHONUSERBASE /app/python
 ---> Using cache
 ---> 6780a1586fdc
Step 3/12 : ADD eggbased-entrypoint /usr/local/sbin/
 ---> Using cache
 ---> d58959161694
Step 4/12 : ADD run-pipcheck /usr/local/bin/
 ---> Using cache
 ---> 1676330f0f58
Step 5/12 : RUN chmod +x /usr/local/bin/run-pipcheck
 ---> Using cache
 ---> cbe9621799ba
Step 6/12 : RUN chmod +x /usr/local/sbin/eggbased-entrypoint &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/start-crawl &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/scrapy-list &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/shub-image-info &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/run-pipcheck
 ---> Using cache
 ---> 661148411a2d
Step 7/12 : ADD requirements.txt /app/requirements.txt
 ---> f586387e3b29
Removing intermediate container 10a6a9e794cb
Step 8/12 : RUN mkdir /app/python && chown nobody:nogroup /app/python
 ---> Running in 47c96d199ba4
 ---> e15cfe130b2f
Removing intermediate container 47c96d199ba4
Step 9/12 : RUN sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --user --no-cache-dir -r /app/requirements.txt
 ---> Running in 7bab963d71d3
Requirement already satisfied (use --upgrade to upgrade): scrapy==1.1.3 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 1))
Collecting boto==2.47.0 (from -r /app/requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/d7/ce/ced61cdf36d09038fc4c88bac1182845489e51a950148f5c1d1a4804cc28/boto-2.47.0-py2.py3-none-any.whl (1.4MB)
Collecting extruct (from -r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/da/73/2ce798fd214c988f06bc78256d06647596cca5632bdde4456a29f3a3fb48/extruct-0.7.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): w3lib==1.14.3 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): jinja2==2.8 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 5))
Collecting awscli==1.15.1 (from -r /app/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/0a/c2/93cc87bb61c62646ec929d1ebb466b5fa47fe8e98ec6971f5d4dbefc6c7a/awscli-1.15.1-py2.py3-none-any.whl (1.2MB)
Requirement already satisfied (use --upgrade to upgrade): idna==2.1 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): PyDispatcher>=2.0.5 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): service-identity in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): parsel>=0.9.3 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1))
Collecting mf2py (from extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/fe/09/116b5b42f5c8cbb1cc43408be9df12fc2bf9fc408dfc9b20311ef4ffbfec/mf2py-1.1.2.tar.gz
Collecting rdflib-jsonld (from extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/ba/48/edaad22fc9de34500699f0c7fc9124385dd425fd18857244f126a6f7df66/rdflib-jsonld-0.4.0.tar.gz (53kB)
Collecting rdflib (from extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/3c/fe/630bacb652680f6d481b9febbb3e2c3869194a1a5fc3401a4a41195a2f8f/rdflib-4.2.2-py3-none-any.whl (344kB)
Requirement already satisfied (use --upgrade to upgrade): MarkupSafe in /usr/local/lib/python3.5/site-packages (from jinja2==2.8->-r /app/requirements.txt (line 5))
Collecting s3transfer<0.2.0,>=0.1.12 (from awscli==1.15.1->-r /app/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/d7/14/2a0004d487464d120c9fb85313a75cd3d71a7506955be458eebfe19a6b1d/s3transfer-0.1.13-py2.py3-none-any.whl (59kB)
Collecting botocore==1.10.1 (from awscli==1.15.1->-r /app/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/ae/e9/304712de6242f0f883dc358adbeee0e05b51d2f538d4bff645dbe11b616e/botocore-1.10.1-py2.py3-none-any.whl (4.2MB)
Requirement already satisfied (use --upgrade to upgrade): rsa<=3.5.0,>=3.1.2 in /usr/local/lib/python3.5/site-packages (from awscli==1.15.1->-r /app/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): colorama<=0.3.7,>=0.2.5 in /usr/local/lib/python3.5/site-packages (from awscli==1.15.1->-r /app/requirements.txt (line 6))
Collecting PyYAML<=3.12,>=3.10 (from awscli==1.15.1->-r /app/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz (253kB)
Requirement already satisfied (use --upgrade to upgrade): docutils>=0.10 in /usr/local/lib/python3.5/site-packages (from awscli==1.15.1->-r /app/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /usr/local/lib/python3.5/site-packages (from service-identity->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): attrs in /usr/local/lib/python3.5/site-packages (from service-identity->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /usr/local/lib/python3.5/site-packages (from service-identity->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=4.0.2 in /usr/local/lib/python3.5/site-packages (from Twisted>=10.0.0->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): cryptography>=1.3 in /usr/local/lib/python3.5/site-packages (from pyOpenSSL->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Collecting html5lib>=1.0.1 (from mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl (117kB)
Collecting requests>=2.18.4 (from mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl (91kB)
Collecting BeautifulSoup4>=4.6.0 (from mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/21/0a/47fdf541c97fd9b6a610cb5fd518175308a7cc60569962e776ac52420387/beautifulsoup4-4.6.3-py3-none-any.whl (90kB)
Collecting isodate (from rdflib->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl (45kB)
Requirement already satisfied (use --upgrade to upgrade): pyparsing in /usr/local/lib/python3.5/site-packages (from rdflib->extruct->-r /app/requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): python-dateutil<2.7.0,>=2.1 in /usr/local/lib/python3.5/site-packages (from botocore==1.10.1->awscli==1.15.1->-r /app/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.5/site-packages (from botocore==1.10.1->awscli==1.15.1->-r /app/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python3.5/site-packages (from zope.interface>=4.0.2->Twisted>=10.0.0->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.4.1 in /usr/local/lib/python3.5/site-packages (from cryptography>=1.3->pyOpenSSL->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Collecting webencodings (from html5lib>=1.0.1->mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.18.4->mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting urllib3<1.24,>=1.21.1 (from requests>=2.18.4->mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB)
Collecting certifi>=2017.4.17 (from requests>=2.18.4->mf2py->extruct->-r /app/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl (147kB)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/local/lib/python3.5/site-packages (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL->scrapy==1.1.3->-r /app/requirements.txt (line 1))
Installing collected packages: boto, webencodings, html5lib, chardet, urllib3, certifi, requests, BeautifulSoup4, mf2py, isodate, rdflib, rdflib-jsonld, extruct, botocore, s3transfer, PyYAML, awscli
  Running setup.py install for mf2py: started
    Running setup.py install for mf2py: finished with status 'done'
  Running setup.py install for rdflib-jsonld: started
    Running setup.py install for rdflib-jsonld: finished with status 'done'
  Running setup.py install for PyYAML: started
    Running setup.py install for PyYAML: finished with status 'done'
Successfully installed BeautifulSoup4 PyYAML awscli-1.10.52 boto-2.42.0 botocore-1.4.42 certifi chardet extruct html5lib isodate mf2py rdflib rdflib-jsonld requests-2.10.0 s3transfer-0.1.1 urllib3 webencodings
You are using pip version 8.1.2, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

 ---> 6228de97ecda
Removing intermediate container 7bab963d71d3
Step 10/12 : COPY *.egg /app/
 ---> 2801677070a2
Removing intermediate container 36861c9d6e30
Step 11/12 : RUN if [ -d "/app/addons_eggs" ]; then rm -f /app/*.dash-addon.egg; fi
 ---> Running in fccb3f29874b
 ---> 1ec4b4e132b6
Removing intermediate container fccb3f29874b
Step 12/12 : ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 ---> Running in 43ccb7fd3f30
 ---> 583810d67794
Removing intermediate container 43ccb7fd3f30
Successfully built 583810d67794
Step 1/3 : FROM alpine:3.5
 ---> 6c6084ed97e5
Step 2/3 : ADD kumo-entrypoint /kumo-entrypoint
 ---> Using cache
 ---> 4e595e8e525b
Step 3/3 : RUN chmod +x /kumo-entrypoint
 ---> Using cache
 ---> 6a61902913ac
Successfully built 6a61902913ac
Entrypoint container is created successfully
>>> Checking python dependencies
Collecting pip==9.0.3
  Downloading https://files.pythonhosted.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
Successfully installed pip-9.0.3
requests 2.19.1 has requirement idna<2.8,>=2.5, but you have idna 2.1.
Warning: Pip checks failed, please fix the conflicts.
{"message": "Dependencies check exit code: 1", "details": "Pip checks failed, please fix the conflicts", "error": "requirements_error"}

{"status": "error", "message": "Requirements error"}
Deploy log location: /var/folders/5p/ys4j0bl16lqgrx5hxyplmykc0000gn/T/shub_deploy_9x0hmgdt.log
Error: Deploy failed: b'{"status": "error", "message": "Requirements error"}'

             My requirements.txt


            scrapy==1.1.3
boto==2.47.0
extruct
w3lib==1.14.3             # via parsel, scrapy
jinja2==2.8

             Please can you help.






             

             

             

             

             

             

             

             

            0 Votes

            nestor

            nestor posted over 6 years ago Admin Best Answer

            I would start by setting a version for your requirements, it's generally bad practice to not specify the version.


            The error is specified in the last few lines of the log:


            requests 2.19.1 has requirement idna<2.8,>=2.5, but you have idna 2.1.


            The stack you're using, scrapy1.1-py3, has idna 2.1 as a requirement: https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt and it conflicts with the requests version being installed. So try setting idna= some version between 2.8 and 2.5


            Or you can also use a newer py3 stack like scrapy1.5-py3.


            0 Votes

            3 Comments

            Sorted by
            nestor

            nestor posted over 6 years ago Admin

            I'm glad it worked :)

            0 Votes

            J

            Jonathan Bowker posted over 6 years ago

            Thanks Nestor,


            I updated to scrapy1.5.1 and requirements.txt now reads:


            scrapy==1.5.1
            boto==2.42.0
            extruct==0.1.0
            w3lib>=1.17.0     # via parsel, scrapy
            jinja2==2.8
            idna==2.6        # via cryptography


            Deploys OK. Thank you

            0 Votes

            nestor

            nestor posted over 6 years ago Admin Answer

            I would start by setting a version for your requirements, it's generally bad practice to not specify the version.


            The error is specified in the last few lines of the log:


            requests 2.19.1 has requirement idna<2.8,>=2.5, but you have idna 2.1.


            The stack you're using, scrapy1.1-py3, has idna 2.1 as a requirement: https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt and it conflicts with the requests version being installed. So try setting idna= some version between 2.8 and 2.5


            Or you can also use a newer py3 stack like scrapy1.5-py3.


            0 Votes

            Login to post a comment

            Preview
            0 of 0