Failure to deploy a spider in Scrapy Cloud

Posted almost 6 years ago by ragno1970

Post a topic

Answered

ragno1970

I am trying to deploy a spider to Scrapy cloud, but I am continuously experiencing requirements problems.

I am using Python 3.7.

My `scrapinghub.yml` file contains these lines:

project: 366695
requirements:
file: requirements.txt

My `requirements.txt` file contains these lines:

asn1crypto==0.24.0
astroid==2.1.0
attrs==18.2.0
Automat==0.7.0
certifi==2018.11.29
cffi==1.11.5
chardet==3.0.4
Click==7.0
constantly==15.1.0
cryptography==2.4.2
cssselect==1.0.3
docker-py==1.10.6
docker-pycreds==0.4.0
hyperlink==18.0.0
idna==2.8
incremental==17.5.0
isort==4.3.4
lazy-object-proxy==1.3.1
lxml==4.2.5
mccabe==0.6.1
parsel==1.5.1
pyasn1==0.4.4
pyasn1-modules==0.2.2
pycparser==2.19
PyDispatcher==2.0.5
PyHamcrest==1.9.0
pylint==2.2.2
pyOpenSSL==18.0.0
PyYAML==3.13
queuelib==1.5.0
requests==2.21.0
retrying==1.3.3
scrapinghub==2.0.3
Scrapy==1.5.1
service-identity==18.1.0
shub==2.8.1
six==1.12.0
tqdm==4.28.1
Twisted==18.9.0
urllib3==1.24.1
w3lib==1.19.0
websocket-client==0.54.0
wrapt==1.10.11
zope.interface==4.6.0
The error message I get is:

Could not find a version that satisfies the requirement astroid==2.1.0 (from -r /app/requirements.txt (line 2)) (from versions: 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.3.8, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5) No matching distribution found for astroid==2.1.0 (from -r /app/requirements.txt (line 2))

Could you please help me?

Thank you

0 Votes

nestor posted almost 6 years ago Admin Best Answer

astroid 2.1.0 is only available on Python 3. Your scrapinghub.yml is not specifying any (py3) stack so it picks the default one which is scrapy-1.3 on Python 2.7. You need to specify a Python 3 stack by adding the suffix -py3 to the stack. See: https://support.scrapinghub.com/support/solutions/articles/22000200402-scrapy-cloud-stacks

0 Votes

1 Comments

nestor posted almost 6 years ago Admin Answer

0 Votes