We have a script using robobrowser that can follow a login form without an issue, but when we add Crawlera (using a Crawlera Session as well), the redirects aren't followed properly and we can't login. The website requires OAuth2 authentication, which might be what is tripping up Crawlera.
Has anyone encountered a similar situation or know how to handle it?
Thanks
0 Votes
s
smartcell posted
over 7 years ago
Best Answer
In case someone runs into the same issue: while using the python library requests, you need to pass the API key into the URL for the proxy rather than use HTTPProxyAuth. Otherwise, the api key isn't properly passed by requests when HTTPS redirects happen.
In case someone runs into the same issue: while using the python library requests, you need to pass the API key into the URL for the proxy rather than use HTTPProxyAuth. Otherwise, the api key isn't properly passed by requests when HTTPS redirects happen.
Hi,
We have a script using robobrowser that can follow a login form without an issue, but when we add Crawlera (using a Crawlera Session as well), the redirects aren't followed properly and we can't login. The website requires OAuth2 authentication, which might be what is tripping up Crawlera.
Has anyone encountered a similar situation or know how to handle it?
Thanks
0 Votes
smartcell posted over 7 years ago Best Answer
In case someone runs into the same issue: while using the python library requests, you need to pass the API key into the URL for the proxy rather than use HTTPProxyAuth. Otherwise, the api key isn't properly passed by requests when HTTPS redirects happen.
This is the way to go:
proxy_auth = "<APIKEY>:" proxies = {"https": "https://{}@{}:{}/".format(proxy_auth, proxy_host, proxy_port), "http": "http://{}@{}:{}/".format(proxy_auth, proxy_host, proxy_port)}
2 Votes
2 Comments
Pablo Hoffman posted over 7 years ago Admin
Thanks for sharing!
0 Votes
smartcell posted over 7 years ago Answer
In case someone runs into the same issue: while using the python library requests, you need to pass the API key into the URL for the proxy rather than use HTTPProxyAuth. Otherwise, the api key isn't properly passed by requests when HTTPS redirects happen.
This is the way to go:
proxy_auth = "<APIKEY>:" proxies = {"https": "https://{}@{}:{}/".format(proxy_auth, proxy_host, proxy_port), "http": "http://{}@{}:{}/".format(proxy_auth, proxy_host, proxy_port)}
2 Votes
Login to post a comment