How to launch a large-scale web scraping project? Find out how LexisNexis did it. Join the webinar on 29th March.Register now

How can we help you today?

Start a new topic
Discussions Zyte Smart Proxy Manager (formerly Crawlera) Technical Help
Answered

Error 400 when using Session ID to use the same IP for a second request

Hi everyone.


Let's see if someone can help me out or, at least, point me in the right direction. I need to use X-Crawlera-Session headers so that I can send a second request to the same slave (IP).


For this, I use X-Crawlera-Session: create in the first request. And get the session ID correctly in the response. I parse the response and use this session id number as a header (X-Crawlera-Session: sessionId) in the second request.


However, the result is always an error 400 (invalid session id). I can't seem to find the error. Been struggling with it all day and looking for information in the forum and docs, but you can hardly find something for PHP.


Thanks everyone in advance!

Best Answer

The problem is that the session ID that you're passing in the second request is incorrect.


You can use the list sessions endpoint to double check that the ID that you're storing the in the variable is correct https://doc.scrapinghub.com/crawlera.html#list-sessions. You also don't need to view the logs to see the session ID in the response, just print out the response headers.

2 people have this question
7 CommentsSorted by Oldest First

Any chance of anyone who has faced the same issue? I would appreciate any help or hint... Thanks in avance!

I am having this same issue.



Most likely you're not storing/capturing the value of the session ID from response header properly.

I would suggest you print/output the response header and the value of your variable for the sessionID and see if they match.


Another reason for 400 error code is the session limit, but it comes with the message user_session_limit. C10 plans have only up to 100 active sessions.

  • Actually, Nestor, I am requestion the creation of the session (X-Crawlera-Session: create) and receive the sessionID correctly. I am outputting both the variable, as parsed by me, along with the response I receive and they do match. 

  • However, when I pass the header 'X-Crawlera-Session: 33176487090’ (note this id is just a sample ID) in the second request as a header (so that I can use the same IP for both requests), then I get this error denying the sessionID and, therefore, I cannot compare it with the one I already received in the first request.
  • I have tried checking in the admin and in the logs (created and downloaded in the CSV option), but they do not appear or do not include the sessionID.


Any idea what the problem can be? Thanks a lot in advance.

Answer

The problem is that the session ID that you're passing in the second request is incorrect.


You can use the list sessions endpoint to double check that the ID that you're storing the in the variable is correct https://doc.scrapinghub.com/crawlera.html#list-sessions. You also don't need to view the logs to see the session ID in the response, just print out the response headers.

Great! Now it's working. I got to see the session ID's by using terminal instead of looking at the logs. It was a problem with the quotes. 


Thanks a lot and again! ;)

No problem.

Login to post a comment
More topics in Technical Help
See all 365 topics