Deployment Failure on Scrapinghub Due to pip Version Conflict (Could Not Find Version Satisfying pip<20.0,>=9.0.3)

Here’s a detailed explanation of everything you’ve done to solve the issue with deploying your Scrapy project to Scrapinghub (Zyte), which you can share with Scrapinghub support.

---

### **Issue Description:**

I'm facing an issue where my Scrapy project deployment to Scrapinghub fails due to a `pip` version conflict. The error message indicates a problem with the version of `pip` during the dependencies check:

```

Collecting pip<20.0,>=9.0.3

WARNING: There're some errors when doing pip-check:

  Could not find a version that satisfies the requirement pip<20.0,>=9.0.3 (from versions: )

No matching distribution found for pip<20.0,>=9.0.3

{"error": "requirements_error", "message": "Dependencies check exit code: 1", "details": "Pip checks failed, please fix the conflicts"}

```

### **Steps Taken to Solve the Issue:**

1. **Created a Virtual Environment:**

   I started by creating a new virtual environment to isolate the project dependencies:

   ```bash

   python3 -m venv venv

   source venv/bin/activate

   ```

2. **Installed Scrapy and Pip Dependencies:**

   I installed Scrapy in the virtual environment and ensured that `pip` was updated:

   ```bash

   pip install scrapy

   pip install --upgrade pip

   ```

3. **Attempted to Deploy to Scrapinghub:**

   I ran the `shub deploy` command to deploy the project:

   ```bash

   shub deploy 770129

   ```

   The deployment failed with the error message related to the `pip` version check, where Scrapinghub expected `pip<20.0,>=9.0.3`.

4. **Created and Updated `requirements.txt`:**

   To resolve the `pip` version conflict, I created a `requirements.txt` file and specified the necessary dependencies, including `pip` version `19.3.1` (which fits the range required by Scrapinghub):

   ```bash

   scrapy==2.11.2

   pip==19.3.1

   ```

   I also ran:

   ```bash

   pip freeze > requirements.txt

   ```

   This file was added to the root of the Scrapy project directory.

5. **Ensured Scrapinghub Uses the `requirements.txt` File:**

   I added the following to my `scrapy.cfg` file to ensure that Scrapinghub uses the `requirements.txt` file during deployment:

   ```ini

   [settings]

   default = demo_scraper.settings

   [deploy]

   url = https://app.zyte.com/api/scrapyd/

   project = 770129

   requirements_file = requirements.txt

   ```

6. **Downgraded `pip` in Virtual Environment:**

   To align with Scrapinghub’s expected version of `pip`, I manually downgraded `pip` in the virtual environment:

   ```bash

   pip install pip==19.3.1

   ```

7. **Re-attempted Deployment:**

   After the changes, I re-attempted deploying the project using:

   ```bash

   shub deploy

   ```

   Unfortunately, the same error persisted:

   ```

   Could not find a version that satisfies the requirement pip<20.0,>=9.0.3

   No matching distribution found for pip<20.0,>=9.0.3

   ```

8. **Checked Logs:**

   I checked the logs generated during the deployment, and the error message consistently pointed to a pip version mismatch.

   Full deploy log (last 30 lines):

   ```

   Step 12/12 : ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

   ---> Running in a80cf4a796f6

   ---> Removed intermediate container a80cf4a796f6

   ---> edb636683fff

   Successfully built edb636683fff

   Successfully tagged i.scrapinghub.com/kumo_project/770129:19

   Step 1/3 : FROM alpine:3.6

   ---> 43773d1dba76

   Step 2/3 : ADD kumo-entrypoint /kumo-entrypoint

   ---> Using cache

   ---> 7376ab0db517

   Step 3/3 : RUN chmod +x /kumo-entrypoint

   ---> Using cache

   ---> 505aff2468ba

   Successfully built 505aff2468ba

   Successfully tagged kumo-entrypoint:latest

   Entrypoint container is created successfully

   >>> Checking python dependencies

   Collecting pip<20.0,>=9.0.3

   WARNING: There're some errors when doing pip-check:

     Could not find a version that satisfies the requirement pip<20.0,>=9.0.3 (from versions: )

   No matching distribution found for pip<20.0,>=9.0.3

   {"error": "requirements_error", "message": "Dependencies check exit code: 1", "details": "Pip checks failed, please fix the conflicts"}

   {"status": "error", "message": "Requirements error"}

   ```

### **Summary of Key Points:**

- I have a fresh virtual environment for the project.

- I created and ensured the use of a `requirements.txt` file with compatible dependencies.

- I downgraded `pip` to a version (`19.3.1`) that fits within Scrapinghub's expected range.

- Despite these steps, the deployment continues to fail due to the `pip` version conflict.

I am unsure why the deployment process still fails with the `pip<20.0,>=9.0.3` error, even though I have followed all the necessary steps to meet the version requirements.

---

You can now send this detailed explanation to Scrapinghub support, and they should be able to help troubleshoot further!