No recent searches
Popular Articles
Sorry! nothing found for
Posted about 6 years ago by dkajtoch
I would like to how to properly use Python requests with Crawlera proxy for encrypted sites. I followed the solution: https://support.scrapinghub.com/support/solutions/articles/22000188407-fetching-https-pages-with-crawlera
and I can fetch sites like https://www.google.com with no problem using 'curl' (command line). I wanted to do a similar thing via Python and requests package, but it does not work. I get 'SubjectAltNameWarning'.
Minimal example:
import requests
PROXY_HOST = 'proxy.crawlera.com'
PROXY_PORT = '8010'
PROXY_AUTH = <my key>
proxies = {
"https": "https://{}@{}:{}/".format(PROXY_AUTH, PROXY_HOST, PROXY_PORT),
"http": "http://{}@{}:{}/".format(PROXY_AUTH, PROXY_HOST, PROXY_PORT)
}
res = requests.get(url='https://www.google.com', proxies=proxies, verify=<absolute-path-to-crawlera-ca.crt>)
0 Votes
dkajtoch posted about 6 years ago Best Answer
Sorry, for a delay. It is working fine. I forgot to put ':' at the end of the key.
Thanks for your interest.
2 Comments
dkajtoch posted about 6 years ago Answer
nestor posted about 6 years ago Admin
Is that the entire output: 'SubjectAltNameWarning'?
Login to post a comment
People who like this
This post will be deleted permanently. Are you sure?
I would like to how to properly use Python requests with Crawlera proxy for encrypted sites. I followed the solution: https://support.scrapinghub.com/support/solutions/articles/22000188407-fetching-https-pages-with-crawlera
and I can fetch sites like https://www.google.com with no problem using 'curl' (command line). I wanted to do a similar thing via Python and requests package, but it does not work. I get 'SubjectAltNameWarning'.
Minimal example:
import requests
PROXY_HOST = 'proxy.crawlera.com'
PROXY_PORT = '8010'
PROXY_AUTH = <my key>
proxies = {
"https": "https://{}@{}:{}/".format(PROXY_AUTH, PROXY_HOST, PROXY_PORT),
"http": "http://{}@{}:{}/".format(PROXY_AUTH, PROXY_HOST, PROXY_PORT)
}
res = requests.get(url='https://www.google.com', proxies=proxies, verify=<absolute-path-to-crawlera-ca.crt>)
0 Votes
dkajtoch posted about 6 years ago Best Answer
Sorry, for a delay. It is working fine. I forgot to put ':' at the end of the key.
Thanks for your interest.
0 Votes
2 Comments
dkajtoch posted about 6 years ago Answer
Sorry, for a delay. It is working fine. I forgot to put ':' at the end of the key.
Thanks for your interest.
0 Votes
nestor posted about 6 years ago Admin
Is that the entire output: 'SubjectAltNameWarning'?
0 Votes
Login to post a comment