Add parser to requirements.txt

Posted almost 8 years ago by Alfonso Moure

Post a topic

Answered

Alfonso Moure

Hi,

I've been following https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud as a guide to include my dependencies in the requirements.txt file, but I'm having some issues.

I'm trying to use urllib.parse to analyse and clean up some URLs. First, I added the module urllib3 to my requirements.txt file as follows:

urllib3==1.22

When I try to deploy the project to Scrapinghub, I see the following error:

ImportError: No module named parse
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}

{"message": "Internal error", "status": "error"}
Deploy log location: /var/folders/sy/5lq7m7v523v7wqkgymgbs8sm0000gn/T/shub_deploy_f9uckshv.log
Error: Deploy failed: b'{"message": "Internal error", "status": "error"}'

Then, I tried to generate the requirements.txt using pip freeze > requirements.txt, just to be sure that I was doing everything the right way. Now I get the following error while trying to deploy the project to Scrapinghub:

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-t95gfR/project/

{"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "details": {"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "code": 1}, "error": "requirements_error"}

{"message": "Requirements error", "status": "error"}
Deploy log location: /var/folders/sy/5lq7m7v523v7wqkgymgbs8sm0000gn/T/shub_deploy_z25sl9b6.log
Error: Deploy failed: b'{"message": "Requirements error", "status": "error"}'

May be this is a stupid question, but... what should I put on my requirements.txt to import urllib.parse? :_)

Thanks!!

0 Votes

thriveni posted almost 8 years ago Admin Best Answer

The error seems to be due to project==1.0 in the requirements.txt.

Collecting project==1.0 (from -r /app/requirements.txt (line 25)) 
Downloading project-1.tar.gz 
Complete output from command python setup.py egg_info: 
---------------------------------------- 
[91mCommand "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-t95gfR/project/

Do you need it for specific purpose? If not, would suggest to remove it from requirements.txt and then deploy.

0 Votes

5 Comments

Alfonso Moure posted almost 8 years ago

Nop, sorry @thriveni, it wasn't that. I have removed project from requiremens.txt, but I still see the same error with urllib.parse when deploying to Scrapinghub:

ImportError: No module named parse
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}

I've just found the problem, and it is kind of stupid and a newbie one... I was having issues with the Python version :_).

Just changed my imports to:

try:
    from urllib.parse import urlparse, parse_qs, urlencode
except ImportError:
    from urlparse import urlparse, parse_qs
    from urllib import urlencode

Issue solved. Sorry for all the inconveniences, thanks for the help!!

0 Votes

thriveni posted almost 8 years ago Admin Answer

The error seems to be due to project==1.0 in the requirements.txt.

Collecting project==1.0 (from -r /app/requirements.txt (line 25)) 
Downloading project-1.tar.gz 
Complete output from command python setup.py egg_info: 
---------------------------------------- 
[91mCommand "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-t95gfR/project/

Do you need it for specific purpose? If not, would suggest to remove it from requirements.txt and then deploy.

0 Votes

Alfonso Moure posted almost 8 years ago

Any ideas about this? :)

Thanks!

0 Votes

Alfonso Moure posted almost 8 years ago

Hi!

Thanks for your reply :).

Here are the contents of my requirements.txt, generated using pip freeze:

apiclient==1.0.3
attrs==16.3.0
certifi==2017.7.27.1
cffi==1.9.1
chardet==3.0.4
click==6.7
constantly==15.1.0
cryptography==1.6
cssselect==1.0.0
docker-py==1.10.6
docker-pycreds==0.2.1
fake-useragent==0.1.7
google-api-python-client==1.6.1
httplib2==0.9.2
hubstorage==0.23.5
idna==2.6
incremental==16.10.1
lxml==3.6.4
numpy==1.11.3
oauth2client==4.0.0
olefile==0.44
pandas==0.19.2
parsel==1.1.0
Pillow==4.2.1
project==1.0
pycparser==2.17
PyDispatcher==2.0.5
pyOpenSSL==16.2.0
python-dateutil==2.6.0
pytz==2016.10
PyYAML==3.12
queuelib==1.4.2
requests==2.18.4
retrying==1.3.3
rsa==3.4.2
scrapinghub==2.0.1
Scrapy==1.2.1
scrapy-fake-useragent==1.1.0
scrapy-splash==0.7
service-identity==16.0.0
shub==2.7.0
simplejson==3.10.0
six==1.10.0
tqdm==4.15.0
treq==17.3.1
Twisted==16.6.0
uritemplate==3.0.0
urllib3==1.22

And these are the contents from my scrapinghub.yum file:

project: 270487
requirements:
  file: ./requirements.txt

The parse module is included in urllib, so I guess it is enough to add it to the requirements.txt.

Thanks!!

0 Votes

thriveni posted almost 8 years ago Admin

Hello,

From the previous Failed deploy the error was related to Module name "parse". You would need to include the Module "Parse" to requirements.txt.

ImportError: No module named parse
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}

Can you share the Requirements.txt that you are using.

0 Votes