Learn all about the latest trends and best practices in data extraction - Join us at Extract SummitGet tickets
Start a new topic

Need help creating setup.py

I have created a myspider1.py which I am able to run as


    $ scrapy runspider myspider1.py


I want to deploy this to ths cloud.


I tried with shub as follows:


    $ shub deploy


and it ran for a while and said:


    Packing version e721061-master

    Created setup.py at /home/xxx/scrapy1/yyy

    Deploying to Scrapy Cloud project "zzz"

    {"status": "ok", "project": zzz, "version": "e721061-master", "spiders": 0}

    Run your spiders at: https://app.scrapinghub.com/p/zzz/


I suspect my setup.py and scrapinghub.yml files are not ok.


    setup.py


    setup(

        name = 'project',

        version = '1.0',

        packages = find_packages(),

        entry_points = {'scrapy': ['settings = test1.settings']},

    )


I don't have a test1.py - so not sure about the settings


Please help!



1 Comment

Hi Sureshv,


Please check the Scrapy Project structure, it should be as below

tutorial/
    scrapy.cfg            # deploy configuration file

    tutorial/             # project's Python module, you'll import your code from here
        __init__.py

        items.py          # project items definition file

        pipelines.py      # project pipelines file

        settings.py       # project settings file

        spiders/          # a directory where you'll later put your spiders
            __init__.py



Ensure that correct folders are mentioned in setup.py and the spider is also under the required folder.


To deploy a Scrapy project to Scrapy Cloud, navigate into the project’s folder and run:



shubdeploy[TARGET]



where [TARGET] is either a project name defined in scrapinghub.yml or a numerical Scrapinghub project ID. If you have configured a default target in your scrapinghub.yml, you can leave out the parameter completely (as you have used).

Login to post a comment