No recent searches
Popular Articles
Sorry! nothing found for
Posted over 6 years ago by Jonathan Bowker
I've been trying for hours to deploy this spider, when trying to do a shub deploy in verbose mode, this is the error message I receive:
$ shub deploy 344151 --verbose Packing version 835d360-master Deploying to Scrapy Cloud project "344151" Login succeeded Building an image: Step 1/12 : FROM scrapinghub/scrapinghub-stack-scrapy:1.1-py3 # Executing 2 build triggers... Step 1/1 : ENV PIP_TRUSTED_HOST $PIP_TRUSTED_HOST PIP_INDEX_URL $PIP_INDEX_URL ---> Using cache Step 1/1 : RUN test -n $APT_PROXY && echo 'Acquire::http::Proxy \"$APT_PROXY\";' >/etc/apt/apt.conf.d/proxy ---> Using cache ---> 5007fbdbcfa4 Step 2/12 : ENV PYTHONUSERBASE /app/python ---> Using cache ---> 6780a1586fdc Step 3/12 : ADD eggbased-entrypoint /usr/local/sbin/ ---> Using cache ---> d58959161694 Step 4/12 : ADD run-pipcheck /usr/local/bin/ ---> Using cache ---> 1676330f0f58 Step 5/12 : RUN chmod +x /usr/local/bin/run-pipcheck ---> Using cache ---> cbe9621799ba Step 6/12 : RUN chmod +x /usr/local/sbin/eggbased-entrypoint && ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/start-crawl && ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/scrapy-list && ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/shub-image-info && ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/run-pipcheck ---> Using cache ---> 661148411a2d Step 7/12 : ADD requirements.txt /app/requirements.txt ---> f586387e3b29 Removing intermediate container 10a6a9e794cb Step 8/12 : RUN mkdir /app/python && chown nobody:nogroup /app/python ---> Running in 47c96d199ba4 ---> e15cfe130b2f Removing intermediate container 47c96d199ba4 Step 9/12 : RUN sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE pip install --user --no-cache-dir -r /app/requirements.txt ---> Running in 7bab963d71d3 Requirement already satisfied (use --upgrade to upgrade): scrapy==1.1.3 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 1)) Collecting boto==2.47.0 (from -r /app/requirements.txt (line 2)) Downloading https://files.pythonhosted.org/packages/d7/ce/ced61cdf36d09038fc4c88bac1182845489e51a950148f5c1d1a4804cc28/boto-2.47.0-py2.py3-none-any.whl (1.4MB) Collecting extruct (from -r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/da/73/2ce798fd214c988f06bc78256d06647596cca5632bdde4456a29f3a3fb48/extruct-0.7.0-py2.py3-none-any.whl Requirement already satisfied (use --upgrade to upgrade): w3lib==1.14.3 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): jinja2==2.8 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 5)) Collecting awscli==1.15.1 (from -r /app/requirements.txt (line 6)) Downloading https://files.pythonhosted.org/packages/0a/c2/93cc87bb61c62646ec929d1ebb466b5fa47fe8e98ec6971f5d4dbefc6c7a/awscli-1.15.1-py2.py3-none-any.whl (1.2MB) Requirement already satisfied (use --upgrade to upgrade): idna==2.1 in /usr/local/lib/python3.5/site-packages (from -r /app/requirements.txt (line 7)) Requirement already satisfied (use --upgrade to upgrade): PyDispatcher>=2.0.5 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): service-identity in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): parsel>=0.9.3 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python3.5/site-packages (from scrapy==1.1.3->-r /app/requirements.txt (line 1)) Collecting mf2py (from extruct->-r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/fe/09/116b5b42f5c8cbb1cc43408be9df12fc2bf9fc408dfc9b20311ef4ffbfec/mf2py-1.1.2.tar.gz Collecting rdflib-jsonld (from extruct->-r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/ba/48/edaad22fc9de34500699f0c7fc9124385dd425fd18857244f126a6f7df66/rdflib-jsonld-0.4.0.tar.gz (53kB) Collecting rdflib (from extruct->-r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/3c/fe/630bacb652680f6d481b9febbb3e2c3869194a1a5fc3401a4a41195a2f8f/rdflib-4.2.2-py3-none-any.whl (344kB) Requirement already satisfied (use --upgrade to upgrade): MarkupSafe in /usr/local/lib/python3.5/site-packages (from jinja2==2.8->-r /app/requirements.txt (line 5)) Collecting s3transfer<0.2.0,>=0.1.12 (from awscli==1.15.1->-r /app/requirements.txt (line 6)) Downloading https://files.pythonhosted.org/packages/d7/14/2a0004d487464d120c9fb85313a75cd3d71a7506955be458eebfe19a6b1d/s3transfer-0.1.13-py2.py3-none-any.whl (59kB) Collecting botocore==1.10.1 (from awscli==1.15.1->-r /app/requirements.txt (line 6)) Downloading https://files.pythonhosted.org/packages/ae/e9/304712de6242f0f883dc358adbeee0e05b51d2f538d4bff645dbe11b616e/botocore-1.10.1-py2.py3-none-any.whl (4.2MB) Requirement already satisfied (use --upgrade to upgrade): rsa<=3.5.0,>=3.1.2 in /usr/local/lib/python3.5/site-packages (from awscli==1.15.1->-r /app/requirements.txt (line 6)) Requirement already satisfied (use --upgrade to upgrade): colorama<=0.3.7,>=0.2.5 in /usr/local/lib/python3.5/site-packages (from awscli==1.15.1->-r /app/requirements.txt (line 6)) Collecting PyYAML<=3.12,>=3.10 (from awscli==1.15.1->-r /app/requirements.txt (line 6)) Downloading https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz (253kB) Requirement already satisfied (use --upgrade to upgrade): docutils>=0.10 in /usr/local/lib/python3.5/site-packages (from awscli==1.15.1->-r /app/requirements.txt (line 6)) Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /usr/local/lib/python3.5/site-packages (from service-identity->scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): attrs in /usr/local/lib/python3.5/site-packages (from service-identity->scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /usr/local/lib/python3.5/site-packages (from service-identity->scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): zope.interface>=4.0.2 in /usr/local/lib/python3.5/site-packages (from Twisted>=10.0.0->scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): cryptography>=1.3 in /usr/local/lib/python3.5/site-packages (from pyOpenSSL->scrapy==1.1.3->-r /app/requirements.txt (line 1)) Collecting html5lib>=1.0.1 (from mf2py->extruct->-r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl (117kB) Collecting requests>=2.18.4 (from mf2py->extruct->-r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl (91kB) Collecting BeautifulSoup4>=4.6.0 (from mf2py->extruct->-r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/21/0a/47fdf541c97fd9b6a610cb5fd518175308a7cc60569962e776ac52420387/beautifulsoup4-4.6.3-py3-none-any.whl (90kB) Collecting isodate (from rdflib->extruct->-r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl (45kB) Requirement already satisfied (use --upgrade to upgrade): pyparsing in /usr/local/lib/python3.5/site-packages (from rdflib->extruct->-r /app/requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): python-dateutil<2.7.0,>=2.1 in /usr/local/lib/python3.5/site-packages (from botocore==1.10.1->awscli==1.15.1->-r /app/requirements.txt (line 6)) Requirement already satisfied (use --upgrade to upgrade): jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.5/site-packages (from botocore==1.10.1->awscli==1.15.1->-r /app/requirements.txt (line 6)) Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python3.5/site-packages (from zope.interface>=4.0.2->Twisted>=10.0.0->scrapy==1.1.3->-r /app/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): cffi>=1.4.1 in /usr/local/lib/python3.5/site-packages (from cryptography>=1.3->pyOpenSSL->scrapy==1.1.3->-r /app/requirements.txt (line 1)) Collecting webencodings (from html5lib>=1.0.1->mf2py->extruct->-r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.18.4->mf2py->extruct->-r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB) Collecting urllib3<1.24,>=1.21.1 (from requests>=2.18.4->mf2py->extruct->-r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB) Collecting certifi>=2017.4.17 (from requests>=2.18.4->mf2py->extruct->-r /app/requirements.txt (line 3)) Downloading https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl (147kB) Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/local/lib/python3.5/site-packages (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL->scrapy==1.1.3->-r /app/requirements.txt (line 1)) Installing collected packages: boto, webencodings, html5lib, chardet, urllib3, certifi, requests, BeautifulSoup4, mf2py, isodate, rdflib, rdflib-jsonld, extruct, botocore, s3transfer, PyYAML, awscli Running setup.py install for mf2py: started Running setup.py install for mf2py: finished with status 'done' Running setup.py install for rdflib-jsonld: started Running setup.py install for rdflib-jsonld: finished with status 'done' Running setup.py install for PyYAML: started Running setup.py install for PyYAML: finished with status 'done' Successfully installed BeautifulSoup4 PyYAML awscli-1.10.52 boto-2.42.0 botocore-1.4.42 certifi chardet extruct html5lib isodate mf2py rdflib rdflib-jsonld requests-2.10.0 s3transfer-0.1.1 urllib3 webencodings You are using pip version 8.1.2, however version 18.0 is available. You should consider upgrading via the 'pip install --upgrade pip' command. ---> 6228de97ecda Removing intermediate container 7bab963d71d3 Step 10/12 : COPY *.egg /app/ ---> 2801677070a2 Removing intermediate container 36861c9d6e30 Step 11/12 : RUN if [ -d "/app/addons_eggs" ]; then rm -f /app/*.dash-addon.egg; fi ---> Running in fccb3f29874b ---> 1ec4b4e132b6 Removing intermediate container fccb3f29874b Step 12/12 : ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ---> Running in 43ccb7fd3f30 ---> 583810d67794 Removing intermediate container 43ccb7fd3f30 Successfully built 583810d67794 Step 1/3 : FROM alpine:3.5 ---> 6c6084ed97e5 Step 2/3 : ADD kumo-entrypoint /kumo-entrypoint ---> Using cache ---> 4e595e8e525b Step 3/3 : RUN chmod +x /kumo-entrypoint ---> Using cache ---> 6a61902913ac Successfully built 6a61902913ac Entrypoint container is created successfully >>> Checking python dependencies Collecting pip==9.0.3 Downloading https://files.pythonhosted.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl (1.4MB) Installing collected packages: pip Successfully installed pip-9.0.3 requests 2.19.1 has requirement idna<2.8,>=2.5, but you have idna 2.1. Warning: Pip checks failed, please fix the conflicts. {"message": "Dependencies check exit code: 1", "details": "Pip checks failed, please fix the conflicts", "error": "requirements_error"} {"status": "error", "message": "Requirements error"} Deploy log location: /var/folders/5p/ys4j0bl16lqgrx5hxyplmykc0000gn/T/shub_deploy_9x0hmgdt.log Error: Deploy failed: b'{"status": "error", "message": "Requirements error"}'
My requirements.txt
scrapy==1.1.3 boto==2.47.0 extruct w3lib==1.14.3 # via parsel, scrapy jinja2==2.8
Please can you help.
0 Votes
nestor posted over 6 years ago Admin Best Answer
I would start by setting a version for your requirements, it's generally bad practice to not specify the version.
The error is specified in the last few lines of the log:
requests 2.19.1 has requirement idna<2.8,>=2.5, but you have idna 2.1.
The stack you're using, scrapy1.1-py3, has idna 2.1 as a requirement: https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt and it conflicts with the requests version being installed. So try setting idna= some version between 2.8 and 2.5
Or you can also use a newer py3 stack like scrapy1.5-py3.
3 Comments
nestor posted over 6 years ago Admin
I'm glad it worked :)
Jonathan Bowker posted over 6 years ago
Thanks Nestor,
I updated to scrapy1.5.1 and requirements.txt now reads:
scrapy==1.5.1 boto==2.42.0extruct==0.1.0w3lib>=1.17.0 # via parsel, scrapyjinja2==2.8idna==2.6 # via cryptography
Deploys OK. Thank you
nestor posted over 6 years ago Admin Answer
Login to post a comment
People who like this
This post will be deleted permanently. Are you sure?
I've been trying for hours to deploy this spider, when trying to do a shub deploy in verbose mode, this is the error message I receive:
My requirements.txt
Please can you help.
0 Votes
nestor posted over 6 years ago Admin Best Answer
I would start by setting a version for your requirements, it's generally bad practice to not specify the version.
The error is specified in the last few lines of the log:
requests 2.19.1 has requirement idna<2.8,>=2.5, but you have idna 2.1.
The stack you're using, scrapy1.1-py3, has idna 2.1 as a requirement: https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt and it conflicts with the requests version being installed. So try setting idna= some version between 2.8 and 2.5
Or you can also use a newer py3 stack like scrapy1.5-py3.
0 Votes
3 Comments
nestor posted over 6 years ago Admin
I'm glad it worked :)
0 Votes
Jonathan Bowker posted over 6 years ago
Thanks Nestor,
I updated to scrapy1.5.1 and requirements.txt now reads:
scrapy==1.5.1
boto==2.42.0
extruct==0.1.0
w3lib>=1.17.0 # via parsel, scrapy
jinja2==2.8
idna==2.6 # via cryptography
Deploys OK. Thank you
0 Votes
nestor posted over 6 years ago Admin Answer
I would start by setting a version for your requirements, it's generally bad practice to not specify the version.
The error is specified in the last few lines of the log:
requests 2.19.1 has requirement idna<2.8,>=2.5, but you have idna 2.1.
The stack you're using, scrapy1.1-py3, has idna 2.1 as a requirement: https://github.com/scrapinghub/scrapinghub-stack-scrapy/blob/branch-1.1-py3/requirements.txt and it conflicts with the requests version being installed. So try setting idna= some version between 2.8 and 2.5
Or you can also use a newer py3 stack like scrapy1.5-py3.
0 Votes
Login to post a comment