Learn all about the latest trends and best practices in data extraction - Join us at Extract SummitGet tickets
Start a new topic

Need help creating setup.py

I have created a myspider1.py which I am able to run as

    $ scrapy runspider myspider1.py

I want to deploy this to ths cloud.

I tried with shub as follows:

    $ shub deploy

and it ran for a while and said:

    Packing version e721061-master

    Created setup.py at /home/xxx/scrapy1/yyy

    Deploying to Scrapy Cloud project "zzz"

    {"status": "ok", "project": zzz, "version": "e721061-master", "spiders": 0}

    Run your spiders at: https://app.scrapinghub.com/p/zzz/

I suspect my setup.py and scrapinghub.yml files are not ok.



        name = 'project',

        version = '1.0',

        packages = find_packages(),

        entry_points = {'scrapy': ['settings = test1.settings']},


I don't have a test1.py - so not sure about the settings

Please help!

1 Comment

Hi Sureshv,

Please check the Scrapy Project structure, it should be as below

    scrapy.cfg            # deploy configuration file

    tutorial/             # project's Python module, you'll import your code from here

        items.py          # project items definition file

        pipelines.py      # project pipelines file

        settings.py       # project settings file

        spiders/          # a directory where you'll later put your spiders

Ensure that correct folders are mentioned in setup.py and the spider is also under the required folder.

To deploy a Scrapy project to Scrapy Cloud, navigate into the project’s folder and run:


where [TARGET] is either a project name defined in scrapinghub.yml or a numerical Scrapinghub project ID. If you have configured a default target in your scrapinghub.yml, you can leave out the parameter completely (as you have used).

Login to post a comment