But when i try to utilize the curl tool to download, it seems doesn't work anymore.
I have notice that you have closed the download viability from dash.scrapinghub.com, which i use to download the data. Is there any way that i still can download the latest data by using curl?
There is still the same error, in fact the very same syntax (with -L) works for me previously. I just wonder that if scrapinghub has withdraw some Curl support function. I just don't know.
Thanks again
nestor
said
about 6 years ago
Answer
I can't reproduce your issue, the URL works fine on my end, no CURL support has been removed.
icong jao
In this article titled "Fetching the latest spider data" (
https://support.scrapinghub.com/support/solutions/articles/22000200409-fetching-latest-spider-data), the answer has provide a convenient way to download the latest data by using a static url.
But when i try to utilize the curl tool to download, it seems doesn't work anymore.
I have notice that you have closed the download viability from dash.scrapinghub.com, which i use to download the data. Is there any way that i still can download the latest data by using curl?
For example, in the past time. I can use
curl -u APIKEY: "https://dash.scrapinghub.com/api/items.csv?
project=PROJECTNUMBER&spider=SPIDERNAME&include_headers=1&fields=FIELDNAME1,FIELDNAME2&apikey=APIKEY"
--compressed -o C:\file.csv
but, it doesn't work now. even i change the coding (only from https://dash. to https://app.) to
curl -u APIKEY: "https://app.scrapinghub.com/api/items.csv?
project=PROJECTNUMBER&spider=SPIDERNAME&include_headers=1&fields=FIELDNAME1,FIELDNAME2&apikey=APIKEY"
--compressed -o C:\file.csv
Only error occurs.
error message:
curl: (35) schannel: next InitializeSecurityContext failed: Unknown error (0x800 92012) -
Could you give me any clue?
Thank you very much
I can't reproduce your issue, the URL works fine on my end, no CURL support has been removed.
You might want to check this out: https://curl.haxx.se/mail/lib-2016-03/0202.html if you are behind a proxy.
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstnestor
You need to instruct curl to follow the redirection with -L
icong jao
Thanks nestor, Even i add -L. i.e.
curl -L -u APIKEY: "https://app.scrapinghub.com/api/items.csv?
project=PROJECTNUMBER&spider=SPIDERNAME&include_headers=1&fields=FIELDNAME1,FIELDNAME2&apikey=APIKEY"
--compressed -o C:\file.csv
There is still the same error, in fact the very same syntax (with -L) works for me previously. I just wonder that if scrapinghub has withdraw some Curl support function. I just don't know.
Thanks again
nestor
I can't reproduce your issue, the URL works fine on my end, no CURL support has been removed.
You might want to check this out: https://curl.haxx.se/mail/lib-2016-03/0202.html if you are behind a proxy.
1 person likes this
icong jao
Problem solved as i disable some proxy and anti-virus software. Thanks for the help.
-
Unable to select Scrapy project in GitHub
-
ScrapyCloud can't call spider?
-
Unhandled error in Deferred
-
Item API - Filtering
-
newbie to web scraping but need data from zillow
-
ValueError: Invalid control character
-
Cancelling account
-
Best Practices
-
Beautifulsoup with ScrapingHub
-
Delete a project in ScrapingHub
See all 446 topics