We have a script using robobrowser that can follow a login form without an issue, but when we add Crawlera (using a Crawlera Session as well), the redirects aren't followed properly and we can't login. The website requires OAuth2 authentication, which might be what is tripping up Crawlera.
Has anyone encountered a similar situation or know how to handle it?
Thanks
Best Answer
s
smartcell
said
about 6 years ago
In case someone runs into the same issue: while using the python library requests, you need to pass the API key into the URL for the proxy rather than use HTTPProxyAuth. Otherwise, the api key isn't properly passed by requests when HTTPS redirects happen.
In case someone runs into the same issue: while using the python library requests, you need to pass the API key into the URL for the proxy rather than use HTTPProxyAuth. Otherwise, the api key isn't properly passed by requests when HTTPS redirects happen.
smartcell
Hi,
We have a script using robobrowser that can follow a login form without an issue, but when we add Crawlera (using a Crawlera Session as well), the redirects aren't followed properly and we can't login. The website requires OAuth2 authentication, which might be what is tripping up Crawlera.
Has anyone encountered a similar situation or know how to handle it?
Thanks
In case someone runs into the same issue: while using the python library requests, you need to pass the API key into the URL for the proxy rather than use HTTPProxyAuth. Otherwise, the api key isn't properly passed by requests when HTTPS redirects happen.
This is the way to go:
proxy_auth = "<APIKEY>:" proxies = {"https": "https://{}@{}:{}/".format(proxy_auth, proxy_host, proxy_port), "http": "http://{}@{}:{}/".format(proxy_auth, proxy_host, proxy_port)}
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstsmartcell
In case someone runs into the same issue: while using the python library requests, you need to pass the API key into the URL for the proxy rather than use HTTPProxyAuth. Otherwise, the api key isn't properly passed by requests when HTTPS redirects happen.
This is the way to go:
proxy_auth = "<APIKEY>:" proxies = {"https": "https://{}@{}:{}/".format(proxy_auth, proxy_host, proxy_port), "http": "http://{}@{}:{}/".format(proxy_auth, proxy_host, proxy_port)}
2 people like this
Pablo Hoffman
Thanks for sharing!
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 381 topics