videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic
Answered

Passing arguments to the spider via jobs API

I want to run a job using jobs API

I am passing the project id and the spider name.

I would like to pass arguments to the spider as well.

  1. How do I pass them via the REST call?
  2. How can the spider consume those arguments? Will they be passed via __init__ ?

Thanks

Avishay


Best Answer

Any parameter not listed here: https://doc.scrapinghub.com/api/jobs.html#run-json, will be treated as an argument. That means you can pass arguments like this:

$ curl -u APIKEY: https://app.scrapinghub.com/api/run.json -d project=123 -d spider=somespider -d spiderarg1=example

me too

Answer

Any parameter not listed here: https://doc.scrapinghub.com/api/jobs.html#run-json, will be treated as an argument. That means you can pass arguments like this:

$ curl -u APIKEY: https://app.scrapinghub.com/api/run.json -d project=123 -d spider=somespider -d spiderarg1=example

1 person likes this

how to capture the argument spiderarg1=example in this spider? 

Can someone tell me how I can access those argument passed to the Job?

Why i continuesly get "invalid value for project" error.

I have passed correct arguments and API


curl -u API: "https://app.scrapinghub.com/api/jobs/list.json?project=tutorial'&spider='quotes'"

The arguments can be captured in the __init__ method and can be used as attributes. Please refer https://docs.scrapy.org/en/latest/topics/spiders.html#spider-arguments to know how to use the arguments. 

hi @bgt-karthi,


The project number should be provided in the URL as

$ curl -u APIKEY: "https://app.scrapinghub.com/api/jobs/list.json?project=123&spider=somespider". Please provide the spidername without the single quotes. 
Login to post a comment