Zyte Support Center

Hi, I made a little change to code (its working on my local scrapy), but while deploying to scrapinghub i am getting this error:

{"status": "error", "message": "Requirements error"}

Here is the full log.

Login succeeded

Building an image:

Step 1/12 : FROM scrapinghub/scrapinghub-stack-scrapy:1.4

# Executing 2 build triggers...

Step 1/1 : ENV PIP_TRUSTED_HOST $PIP_TRUSTED_HOST PIP_INDEX_URL $PIP_INDEX_URL

 ---> Using cache

Step 1/1 : RUN test -n $APT_PROXY && echo 'Acquire::http::Proxy \"$APT_PROXY\";'  >/etc/apt/apt.conf.d/proxy

 ---> Using cache

 ---> 24d6293b6b61

Step 2/12 : ENV PYTHONUSERBASE /app/python

 ---> Using cache

 ---> 61cdf4ed7547

Step 3/12 : ADD eggbased-entrypoint /usr/local/sbin/

 ---> Using cache

 ---> 92017487150e

Step 4/12 : ADD run-pipcheck /usr/local/bin/

 ---> Using cache

 ---> a8a333f93303

Step 5/12 : RUN chmod +x /usr/local/bin/run-pipcheck

 ---> Using cache

 ---> b1bd6a21e692

Step 6/12 : RUN chmod +x /usr/local/sbin/eggbased-entrypoint &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/start-crawl &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/scrapy-list &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/shub-image-info &&     ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/run-pipcheck

 ---> Using cache

 ---> 24fb77a956ac

Step 7/12 : ADD requirements.txt /app/requirements.txt

 ---> Using cache

 ---> be3e85f65e23

Step 8/12 : RUN mkdir /app/python && chown nobody:nogroup /app/python

 ---> Using cache

 ---> f39f9e835f37

Step 9/12 : RUN sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE  pip install --user --no-cache-dir -r /app/requirements.txt

 ---> Running in 640f6ab4299b

Collecting pdfminer==20140328 (from -r /app/requirements.txt (line 1))

 Downloading https://files.pythonhosted.org/packages/57/4f/e1df0437858188d2d36466a7bb89aa024d252bd0b7e3ba90cbc567c6c0b8/pdfminer-20140328.tar.gz (4.1MB)

Collecting pendulum==2.0.2 (from -r /app/requirements.txt (line 2))

 Downloading https://files.pythonhosted.org/packages/4a/97/c288b8c98e49b9e1ec67b28ada3f39b7bd786be11ca119244e2aca3a5cf4/pendulum-2.0.2.tar.gz (75kB)

 Installing build dependencies: started

 Installing build dependencies: finished with status 'error'

 Complete output from command /usr/local/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-OJLJb2 --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=38.2.5 wheel:

 The directory '/root/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

 The directory '/root/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

 Collecting setuptools>=38.2.5

 Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/root/.cache/pip/wheels/a3/63/e3/a02e38a020e587e1cb7b263c1f26ebb218fd3d87f77584662b'

 Consider using the `--user` option or check the permissions.

 ----------------------------------------

Command "/usr/local/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-OJLJb2 --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=38.2.5 wheel" failed with error code 1 in None

{"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "details": {"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE     pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "code": 1}, "error": "requirements_error"}

Requirements  file:

pdfminer==20140328
pendulum==2.0.2
parsel==1.4.0
openpyxl==2.5.4

project: {PROJ NUMBER} (HIDDEN)
requirements:
  file: requirements.txt

More info about bug: All requirements are ok except pendulum.

After removing pendulum from requirements deployment process is finished, but there is import error after (some spiders are using pendulum). So problem is not solved, because we still need pendulum for our spiders. Before weekend it was ok, we're using pendulum since last year.