Then, I tried to generate the requirements.txt using pip freeze > requirements.txt, just to be sure that I was doing everything the right way. Now I get the following error while trying to deploy the project to Scrapinghub:
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-t95gfR/project/
{"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "details": {"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "code": 1}, "error": "requirements_error"}
{"message": "Requirements error", "status": "error"}
Deploy log location: /var/folders/sy/5lq7m7v523v7wqkgymgbs8sm0000gn/T/shub_deploy_z25sl9b6.log
Error: Deploy failed: b'{"message": "Requirements error", "status": "error"}'
May be this is a stupid question, but... what should I put on my requirements.txt to import urllib.parse? :_)
Thanks!!
Best Answer
t
thriveni
said
over 6 years ago
The error seems to be due to project==1.0 in the requirements.txt.
Collecting project==1.0 (from -r /app/requirements.txt (line 25))
Downloading project-1.tar.gz
Complete output from command python setup.py egg_info:
----------------------------------------
[91mCommand "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-t95gfR/project/
Do you need it for specific purpose? If not, would suggest to remove it from requirements.txt and then deploy.
The parse module is included in urllib, so I guess it is enough to add it to the requirements.txt.
Thanks!!
Alfonso Moure
said
over 6 years ago
Any ideas about this? :)
Thanks!
thriveni
said
over 6 years ago
Answer
The error seems to be due to project==1.0 in the requirements.txt.
Collecting project==1.0 (from -r /app/requirements.txt (line 25))
Downloading project-1.tar.gz
Complete output from command python setup.py egg_info:
----------------------------------------
[91mCommand "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-t95gfR/project/
Do you need it for specific purpose? If not, would suggest to remove it from requirements.txt and then deploy.
Alfonso Moure
said
over 6 years ago
Nop, sorry @thriveni, it wasn't that. I have removed project from requiremens.txt, but I still see the same error with urllib.parse when deploying to Scrapinghub:
ImportError: No module named parse
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}
I've just found the problem, and it is kind of stupid and a newbie one... I was having issues with the Python version :_).
Just changed my imports to:
try:
from urllib.parse import urlparse, parse_qs, urlencode
except ImportError:
from urlparse import urlparse, parse_qs
from urllib import urlencode
Issue solved. Sorry for all the inconveniences, thanks for the help!!
Alfonso Moure
Hi,
I've been following https://support.scrapinghub.com/support/solutions/articles/22000200400-deploying-python-dependencies-for-your-projects-in-scrapy-cloud as a guide to include my dependencies in the requirements.txt file, but I'm having some issues.
I'm trying to use urllib.parse to analyse and clean up some URLs. First, I added the module urllib3 to my requirements.txt file as follows:
When I try to deploy the project to Scrapinghub, I see the following error:
Then, I tried to generate the requirements.txt using pip freeze > requirements.txt, just to be sure that I was doing everything the right way. Now I get the following error while trying to deploy the project to Scrapinghub:
May be this is a stupid question, but... what should I put on my requirements.txt to import urllib.parse? :_)
Thanks!!
The error seems to be due to project==1.0 in the requirements.txt.
Do you need it for specific purpose? If not, would suggest to remove it from requirements.txt and then deploy.
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstthriveni
Hello,
From the previous Failed deploy the error was related to Module name "parse". You would need to include the Module "Parse" to requirements.txt.
Can you share the Requirements.txt that you are using.
Alfonso Moure
Hi!
Thanks for your reply :).
Here are the contents of my requirements.txt, generated using pip freeze:
And these are the contents from my scrapinghub.yum file:
The parse module is included in urllib, so I guess it is enough to add it to the requirements.txt.
Thanks!!
Alfonso Moure
Any ideas about this? :)
Thanks!
thriveni
The error seems to be due to project==1.0 in the requirements.txt.
Do you need it for specific purpose? If not, would suggest to remove it from requirements.txt and then deploy.
Alfonso Moure
Nop, sorry @thriveni, it wasn't that. I have removed project from requiremens.txt, but I still see the same error with urllib.parse when deploying to Scrapinghub:
I've just found the problem, and it is kind of stupid and a newbie one... I was having issues with the Python version :_).
Just changed my imports to:
Issue solved. Sorry for all the inconveniences, thanks for the help!!
-
Unable to select Scrapy project in GitHub
-
ScrapyCloud can't call spider?
-
Unhandled error in Deferred
-
Item API - Filtering
-
newbie to web scraping but need data from zillow
-
ValueError: Invalid control character
-
Cancelling account
-
Best Practices
-
Beautifulsoup with ScrapingHub
-
Delete a project in ScrapingHub
See all 453 topics