and I have tried the suggestions listed there but they don't work.
Running locally, the command line arguments are passed to the from_crawler method as expected but not when running from Scraping Hub.
What can I do to resolve this?
Best Answer
t
thriveni
said
over 3 years ago
Hello,
You can pass Arguments while running the Jon through UI (Arguments section of the Run pop-up) as shown below or cURL data option if running a job via API.
You can pass Arguments while running the Jon through UI (Arguments section of the Run pop-up) as shown below or cURL data option if running a job via API.
1 person likes this
M
Matthias Kurwig
said
over 3 years ago
Thank you very much! The arguments are passed correctly now.
However, I'm still confused as to how to access the settings defined on the spider settings page.
Matthias Kurwig
Hi, I am having a similar issue as the one described here: https://support.scrapinghub.com/support/discussions/topics/22000009043
and I have tried the suggestions listed there but they don't work.
Running locally, the command line arguments are passed to the from_crawler method as expected but not when running from Scraping Hub.
What can I do to resolve this?
Hello,
You can pass Arguments while running the Jon through UI (Arguments section of the Run pop-up) as shown below or cURL data option if running a job via API.
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstthriveni
Hello,
You can pass Arguments while running the Jon through UI (Arguments section of the Run pop-up) as shown below or cURL data option if running a job via API.
1 person likes this
Matthias Kurwig
Thank you very much! The arguments are passed correctly now.
However, I'm still confused as to how to access the settings defined on the spider settings page.
thriveni
Glad to know that Arguments worked :)
For accessing the Spider settings, you would need to navigate to setting page of the spider like example https://app.scrapinghub.com/p/444534/3/settings and then use the (+),(-) and add or modify the settings. Please refer https://support.scrapinghub.com/a/solutions/articles/22000200670 to know about spider settings page.
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 389 topics