How to launch a large-scale web scraping project? Find out how LexisNexis did it. Join the webinar on 29th March.Register now

How can we help you today?

Start a new topic
Discussions Scrapy Cloud Technical Help
Answered

Failure to deploy a spider in Scrapy Cloud

I am trying to deploy a spider to Scrapy cloud, but I am continuously experiencing requirements problems.

I am using Python 3.7.

My `scrapinghub.yml` file contains these lines: 

project: 366695
requirements:
file: requirements.txt

My `requirements.txt` file contains these lines:

asn1crypto==0.24.0
astroid==2.1.0
attrs==18.2.0
Automat==0.7.0
certifi==2018.11.29
cffi==1.11.5
chardet==3.0.4
Click==7.0
constantly==15.1.0
cryptography==2.4.2
cssselect==1.0.3
docker-py==1.10.6
docker-pycreds==0.4.0
hyperlink==18.0.0
idna==2.8
incremental==17.5.0
isort==4.3.4
lazy-object-proxy==1.3.1
lxml==4.2.5
mccabe==0.6.1
parsel==1.5.1
pyasn1==0.4.4
pyasn1-modules==0.2.2
pycparser==2.19
PyDispatcher==2.0.5
PyHamcrest==1.9.0
pylint==2.2.2
pyOpenSSL==18.0.0
PyYAML==3.13
queuelib==1.5.0
requests==2.21.0
retrying==1.3.3
scrapinghub==2.0.3
Scrapy==1.5.1
service-identity==18.1.0
shub==2.8.1
six==1.12.0
tqdm==4.28.1
Twisted==18.9.0
urllib3==1.24.1
w3lib==1.19.0
websocket-client==0.54.0
wrapt==1.10.11
zope.interface==4.6.0
The error message I get is:

Could not find a version that satisfies the requirement astroid==2.1.0 (from -r /app/requirements.txt (line 2)) (from versions: 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.3.8, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5) No matching distribution found for astroid==2.1.0 (from -r /app/requirements.txt (line 2))

Could you please help me?

Thank you
Best Answer

astroid 2.1.0 is only available on Python 3. Your scrapinghub.yml is not specifying any (py3) stack so it picks the default one which is scrapy-1.3 on Python 2.7. You need to specify a Python 3 stack by adding the suffix -py3 to the stack. See: https://support.scrapinghub.com/support/solutions/articles/22000200402-scrapy-cloud-stacks

1 Comment
Answer

astroid 2.1.0 is only available on Python 3. Your scrapinghub.yml is not specifying any (py3) stack so it picks the default one which is scrapy-1.3 on Python 2.7. You need to specify a Python 3 stack by adding the suffix -py3 to the stack. See: https://support.scrapinghub.com/support/solutions/articles/22000200402-scrapy-cloud-stacks

Login to post a comment
More topics in Technical Help
See all 434 topics