How can we help you today?

Start a new topic
Discussions Scrapy Cloud Technical Help
Answered

Passing arguments to the spider via jobs API

I want to run a job using jobs API. 

I am passing the project id and the spider name.

I would like to pass arguments to the spider as well.

  1. How do I pass them via the REST call?
  2. How can the spider consume those arguments? Will they be passed via __init__ ?

Thanks

Avishay

Best Answer

Any parameter not listed here: https://doc.scrapinghub.com/api/jobs.html#run-json, will be treated as an argument. That means you can pass arguments like this:

$ curl -u APIKEY: https://app.scrapinghub.com/api/run.json -d project=123 -d spider=somespider -d spiderarg1=example
7 CommentsSorted by Newest First

hi @bgt-karthi,


The project number should be provided in the URL as

$ curl -u APIKEY: "https://app.scrapinghub.com/api/jobs/list.json?project=123&spider=somespider". Please provide the spidername without the single quotes.

The arguments can be captured in the __init__ method and can be used as attributes. Please refer https://docs.scrapy.org/en/latest/topics/spiders.html#spider-arguments to know how to use the arguments. 

Why i continuesly get "invalid value for project" error.

I have passed correct arguments and API


curl -u API: "https://app.scrapinghub.com/api/jobs/list.json?project=tutorial'&spider='quotes'"

Can someone tell me how I can access those argument passed to the Job?

how to capture the argument spiderarg1=example in this spider? 

Answer

Any parameter not listed here: https://doc.scrapinghub.com/api/jobs.html#run-json, will be treated as an argument. That means you can pass arguments like this:

$ curl -u APIKEY: https://app.scrapinghub.com/api/run.json -d project=123 -d spider=somespider -d spiderarg1=example
1 person likes this

me too

Login to post a comment
More topics in Technical Help
See all 453 topics