videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic

Get auth token from javascript response script

I need help about this strange scenario.


- calling site http://www.mysite,com/index.php , I can get this information from tag script  

https://www.mysite.com/anotherpage.php?key=ABCDFG

- using the key , I have  to call this endpoint 

https://www.mysite.com/login.php?key=ABCD 

 for retrieving the SessionID which is inside a javascript 

-- omitted
    
private._sessID='MYSESSIONID';

-- omitted

using the sessionId and performing right POST actions , I can navigate inside all pages I need and that is a good thing :)


My issue : 

I'm able to simulate all steps using scrapy shell and regEx (and all work fine) but I don't know how to manage these steps inside a spider before starting my extraction.


Could you help me in some way ?


Thanks in advance

Luca

Login to post a comment