videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic
Answered

How execute api run.json in browser or aps classic?

 I wish execute curl -u curl -u APIKEY: https://app.scrapinghub.com/api/run.json -d project=284337 -d spider=even in browser or method request asp classic.


I tried, but not fuction

curl https://app.scrapinghub.com/api/run.json?apikey=<APIKEY>&project=284337&spider=event


Best Answer
I was confused, but I got it.
My procedure was,

Run with curl with -v on the terminal
curl -v -u APIKEY: https://app.scrapinghub.com/api/run.json -d project = 284337 -d spider = event

and then I could see the send header,

* Using HTTP2, server supports multi-use
* Connection state changed (HTTP / 2 confirmed)
* Copying HTTP / 2 data in stream buffer to connection buffer after upgrade: len = 0
* Server auth using Basic with user 'APIKEY'
* Using Stream ID: 1 (easy handle 0x5593288dada0)
> POST /api/run.json HTTP / 1.1
> Host: app.scrapinghub.com
> Authorization: Basic APIKEY Converted
> User-Agent: curl / 7.52.1
> Accept: * / *
> Content-Length: 39
> Content-Type: application / x-www-form-urlencoded


 

<% 
dim data
data = "project=284337"
data = data & "&spider=event"

Set obj = CreateObject("MSXML2.ServerXMLHTTP")
obj.Open "POST", "https://app.scrapinghub.com/api/run.json", False 
obj.SetRequestHeader "Content-Type", "application/x-www-form-urlencoded"
obj.SetRequestHeader "Authorization", "Basic APIKEY Converted"
obj.Send data
Response.Write obj.ResponseText
Set obj = Nothing 
%>

 

 


curl is not executed from a browser, it is a command line tool executed from a console or a terminal.

That API request is a POST request, why do you need to run it from a browser anyways?


Btw, this is a public forum so I've just edited your post and erased your APIKey from it, but I would suggest you generate a new one from your Account Settings.

I Know curl is command terminal, I need requests get in asp classic for get item json after run spider.

 

To run the spider you need to pass the parameters as data and the method is a POST request.


To the get the item you can send a GET request like so: https://storage.scrapinghub.com/items/projectid/spidernumberid/jobid (don't forget to include the APIKey)

Thank you, i will try

 

I tried, not sucess, how I can pass apikey in params,

apikey=APIKEY ? or send how username ?


 

<% 
dim data
data = "apikey=APIKEY"
data = data & "&project=284337"
data = data & "&spider=event"
Set obj = CreateObject("MSXML2.ServerXMLHTTP")
obj.Open "POST", "https://app.scrapinghub.com/api/run.json", False, 
obj.SetRequestHeader "Content-Type", "application/json"
obj.Send data
Response.Write obj.ResponseText
Set obj = Nothing 
%>

 

 

Answer
I was confused, but I got it.
My procedure was,

Run with curl with -v on the terminal
curl -v -u APIKEY: https://app.scrapinghub.com/api/run.json -d project = 284337 -d spider = event

and then I could see the send header,

* Using HTTP2, server supports multi-use
* Connection state changed (HTTP / 2 confirmed)
* Copying HTTP / 2 data in stream buffer to connection buffer after upgrade: len = 0
* Server auth using Basic with user 'APIKEY'
* Using Stream ID: 1 (easy handle 0x5593288dada0)
> POST /api/run.json HTTP / 1.1
> Host: app.scrapinghub.com
> Authorization: Basic APIKEY Converted
> User-Agent: curl / 7.52.1
> Accept: * / *
> Content-Length: 39
> Content-Type: application / x-www-form-urlencoded


 

<% 
dim data
data = "project=284337"
data = data & "&spider=event"

Set obj = CreateObject("MSXML2.ServerXMLHTTP")
obj.Open "POST", "https://app.scrapinghub.com/api/run.json", False 
obj.SetRequestHeader "Content-Type", "application/x-www-form-urlencoded"
obj.SetRequestHeader "Authorization", "Basic APIKEY Converted"
obj.Send data
Response.Write obj.ResponseText
Set obj = Nothing 
%>

 

 

APIKEY Converted = EncodeBase64(APIKEY)

 

right, apikey is not a parameter in the Jobs API

Login to post a comment