Need help creating setup.py

Start a new topic

Need help creating setup.py

sureshvv

started a topic over 5 years ago

I have created a myspider1.py which I am able to run as

$ scrapy runspider myspider1.py

I want to deploy this to ths cloud.

I tried with shub as follows:

$ shub deploy

and it ran for a while and said:

Packing version e721061-master

Created setup.py at /home/xxx/scrapy1/yyy

Deploying to Scrapy Cloud project "zzz"

{"status": "ok", "project": zzz, "version": "e721061-master", "spiders": 0}

Run your spiders at: https://app.scrapinghub.com/p/zzz/

I suspect my setup.py and scrapinghub.yml files are not ok.

setup.py

setup(

name = 'project',

version = '1.0',

packages = find_packages(),

entry_points = {'scrapy': ['settings = test1.settings']},

)

I don't have a test1.py - so not sure about the settings

Please help!

1 Comment

thriveni

said over 5 years ago

Hi Sureshv,

Please check the Scrapy Project structure, it should be as below

tutorial/
    scrapy.cfg            # deploy configuration file

    tutorial/             # project's Python module, you'll import your code from here
        __init__.py

        items.py          # project items definition file

        pipelines.py      # project pipelines file

        settings.py       # project settings file

        spiders/          # a directory where you'll later put your spiders
            __init__.py

Ensure that correct folders are mentioned in setup.py and the spider is also under the required folder.

To deploy a Scrapy project to Scrapy Cloud, navigate into the project’s folder and run:

shubdeploy[TARGET]

where [TARGET] is either a project name defined in scrapinghub.yml or a numerical Scrapinghub project ID. If you have configured a default target in your scrapinghub.yml, you can leave out the parameter completely (as you have used).

Zyte Support Center

How can we help you today?