I would like to how to properly use Python requests with Crawlera proxy for encrypted sites. I followed the solution: https://support.scrapinghub.com/support/solutions/articles/22000188407-fetching-https-pages-with-crawlera
and I can fetch sites like https://www.google.com with no problem using 'curl' (command line). I wanted to do a similar thing via Python and requests package, but it does not work. I get 'SubjectAltNameWarning'.
Minimal example:
import requests
PROXY_HOST = 'proxy.crawlera.com'
PROXY_PORT = '8010'
PROXY_AUTH = <my key>
proxies = {
"https": "https://{}@{}:{}/".format(PROXY_AUTH, PROXY_HOST, PROXY_PORT),
"http": "http://{}@{}:{}/".format(PROXY_AUTH, PROXY_HOST, PROXY_PORT)
}
res = requests.get(url='https://www.google.com', proxies=proxies, verify=<absolute-path-to-crawlera-ca.crt>)
Sorry, for a delay. It is working fine. I forgot to put ':' at the end of the key.
Thanks for your interest.
Is that the entire output: 'SubjectAltNameWarning'?
dkajtoch
I would like to how to properly use Python requests with Crawlera proxy for encrypted sites. I followed the solution: https://support.scrapinghub.com/support/solutions/articles/22000188407-fetching-https-pages-with-crawlera
and I can fetch sites like https://www.google.com with no problem using 'curl' (command line). I wanted to do a similar thing via Python and requests package, but it does not work. I get 'SubjectAltNameWarning'.
Minimal example:
import requests
PROXY_HOST = 'proxy.crawlera.com'
PROXY_PORT = '8010'
PROXY_AUTH = <my key>
proxies = {
"https": "https://{}@{}:{}/".format(PROXY_AUTH, PROXY_HOST, PROXY_PORT),
"http": "http://{}@{}:{}/".format(PROXY_AUTH, PROXY_HOST, PROXY_PORT)
}
res = requests.get(url='https://www.google.com', proxies=proxies, verify=<absolute-path-to-crawlera-ca.crt>)
Sorry, for a delay. It is working fine. I forgot to put ':' at the end of the key.
Thanks for your interest.
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstnestor
Is that the entire output: 'SubjectAltNameWarning'?
dkajtoch
Sorry, for a delay. It is working fine. I forgot to put ':' at the end of the key.
Thanks for your interest.
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 399 topics