videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic
Answered

Failure to deploy a spider in Scrapy Cloud

I am trying to deploy a spider to Scrapy cloud, but I am continuously experiencing requirements problems.

I am using Python 3.7.

My `scrapinghub.yml` file contains these lines: 

project: 366695
requirements:
file: requirements.txt

My `requirements.txt` file contains these lines:

asn1crypto==0.24.0
astroid==2.1.0
attrs==18.2.0
Automat==0.7.0
certifi==2018.11.29
cffi==1.11.5
chardet==3.0.4
Click==7.0
constantly==15.1.0
cryptography==2.4.2
cssselect==1.0.3
docker-py==1.10.6
docker-pycreds==0.4.0
hyperlink==18.0.0
idna==2.8
incremental==17.5.0
isort==4.3.4
lazy-object-proxy==1.3.1
lxml==4.2.5
mccabe==0.6.1
parsel==1.5.1
pyasn1==0.4.4
pyasn1-modules==0.2.2
pycparser==2.19
PyDispatcher==2.0.5
PyHamcrest==1.9.0
pylint==2.2.2
pyOpenSSL==18.0.0
PyYAML==3.13
queuelib==1.5.0
requests==2.21.0
retrying==1.3.3
scrapinghub==2.0.3
Scrapy==1.5.1
service-identity==18.1.0
shub==2.8.1
six==1.12.0
tqdm==4.28.1
Twisted==18.9.0
urllib3==1.24.1
w3lib==1.19.0
websocket-client==0.54.0
wrapt==1.10.11
zope.interface==4.6.0
The error message I get is:

Could not find a version that satisfies the requirement astroid==2.1.0 (from -r /app/requirements.txt (line 2)) (from versions: 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.3.8, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5) No matching distribution found for astroid==2.1.0 (from -r /app/requirements.txt (line 2))

Could you please help me?

Thank you

Best Answer

astroid 2.1.0 is only available on Python 3. Your scrapinghub.yml is not specifying any (py3) stack so it picks the default one which is scrapy-1.3 on Python 2.7. You need to specify a Python 3 stack by adding the suffix -py3 to the stack. See: https://support.scrapinghub.com/support/solutions/articles/22000200402-scrapy-cloud-stacks

1 Comment

Answer

astroid 2.1.0 is only available on Python 3. Your scrapinghub.yml is not specifying any (py3) stack so it picks the default one which is scrapy-1.3 on Python 2.7. You need to specify a Python 3 stack by adding the suffix -py3 to the stack. See: https://support.scrapinghub.com/support/solutions/articles/22000200402-scrapy-cloud-stacks

Login to post a comment