Passing arguments to the spider via jobs API

Posted over 7 years ago by Avishay Balderman

Post a topic

Answered

Avishay Balderman

I want to run a job using jobs API.

I am passing the project id and the spider name.

I would like to pass arguments to the spider as well.

How do I pass them via the REST call?
How can the spider consume those arguments? Will they be passed via __init__ ?

Thanks

Avishay

0 Votes

nestor posted over 7 years ago Admin Best Answer

Any parameter not listed here: https://doc.scrapinghub.com/api/jobs.html#run-json, will be treated as an argument. That means you can pass arguments like this:

$ curl -u APIKEY: https://app.scrapinghub.com/api/run.json -d project=123 -d spider=somespider -d spiderarg1=example

1 Votes

7 Comments

nestor posted over 7 years ago Admin Answer

Any parameter not listed here: https://doc.scrapinghub.com/api/jobs.html#run-json, will be treated as an argument. That means you can pass arguments like this:

$ curl -u APIKEY: https://app.scrapinghub.com/api/run.json -d project=123 -d spider=somespider -d spiderarg1=example

1 Votes

Marcos Machado posted over 7 years ago

me too

0 Votes

saptadiparc posted over 6 years ago

how to capture the argument spiderarg1=example in this spider?

0 Votes

Navneet Verma posted almost 5 years ago

Can someone tell me how I can access those argument passed to the Job?

0 Votes

bgt-karthi posted almost 5 years ago

Why i continuesly get "invalid value for project" error.

I have passed correct arguments and API

curl -u API: "https://app.scrapinghub.com/api/jobs/list.json?project=tutorial'&spider='quotes'"

0 Votes

thriveni posted over 4 years ago Admin

The arguments can be captured in the __init__ method and can be used as attributes. Please refer https://docs.scrapy.org/en/latest/topics/spiders.html#spider-arguments to know how to use the arguments.

0 Votes

thriveni posted over 4 years ago Admin

hi @bgt-karthi,

The project number should be provided in the URL as

$ curl -u APIKEY: "https://app.scrapinghub.com/api/jobs/list.json?project=123&spider=somespider". Please provide the spidername without the single quotes.

0 Votes