error running storage.scrapinghub.com/items

Posted almost 6 years ago by Marcos Machado

Post a topic

Un Answered

Marcos Machado

Hi!

I't tryng put in code the curl command below:

curl -u <APIKEY> "https://storage.scrapinghub.com/items/010140/1/24870"

on golang

req,err := http.NewRequest("GET", "https://storage.scrapinghub.com/items/"+jobId, nil)

req.setBasicAuth(key,"")

req.Do()

I got the Unauthorized message

I'm using golang to send a job request and consult the job status (both work).. but to get itens from the job finished isn't work.

I changed GET to POST, but no success

0 Votes

3 Comments

nestor posted almost 6 years ago Admin

Try setting Basic Auth as a header instead, with base64 encoded "apikey:" as opposed to SetBasicAuth. Without more details about what how you're capturing the value of jobID and a real sample ID, I can't provide more information.
I can only infer what I mentioned in my last message: "It seems like you're getting a redirect and GoLang doesn't propagate SetBasicAuth to the followed redirect."

0 Votes

Marcos Machado posted almost 6 years ago

I'm putting the id of the job ( xxx/x/xxxx). The var jobId works when I call:

exec.Command("curl", "-u",key, "https://storage.scrapinghub.com/items/"+jobId)

0 Votes

nestor posted almost 6 years ago Admin

This "https://storage.scrapinghub.com/items/010140/1/24870" doesn't seem to be a valid job id.

What are you putting as jobId in your GoLang script? It seems like you're getting a redirect and GoLang doesn't propagate BasicAuth to the followed redirect.

1 Votes