Spider arguments ignored

Posted almost 5 years ago by Matthias Kurwig

Post a topic

Answered

Matthias Kurwig

Hi, I am having a similar issue as the one described here: https://support.scrapinghub.com/support/discussions/topics/22000009043

and I have tried the suggestions listed there but they don't work.

Running locally, the command line arguments are passed to the from_crawler method as expected but not when running from Scraping Hub.

What can I do to resolve this?

0 Votes

thriveni posted almost 5 years ago Admin Best Answer

Hello,

You can pass Arguments while running the Jon through UI (Arguments section of the Run pop-up) as shown below or cURL data option if running a job via API.

1 Votes

3 Comments

thriveni posted almost 5 years ago Admin

Glad to know that Arguments worked :)

For accessing the Spider settings, you would need to navigate to setting page of the spider like example https://app.scrapinghub.com/p/444534/3/settings and then use the (+),(-) and add or modify the settings. Please refer https://support.scrapinghub.com/a/solutions/articles/22000200670 to know about spider settings page.

0 Votes

Matthias Kurwig posted almost 5 years ago

Thank you very much! The arguments are passed correctly now.

However, I'm still confused as to how to access the settings defined on the spider settings page.

0 Votes

thriveni posted almost 5 years ago Admin Answer

Hello,

You can pass Arguments while running the Jon through UI (Arguments section of the Run pop-up) as shown below or cURL data option if running a job via API.

1 Votes