videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic

Could not obtain new session even after retry

I have set up the Zyte Smart Proxy Manager and also downloaded the certitficate from "https://docs.zyte.com/_static/zyte-smartproxy-ca.crt" URL and added to my key chain, marked it as trusted but whenever i'm trying to access any URL from selenium chromedriver. It is not working and giving me invalid certificate error. 


Also I'm getting this warning into my logs - 


WARN[0136] Could not obtain new session even after retry 




Please let me know if there is any other URL for the certificate? Or i've missed something?


Hello,


Please ensure that you have downloaded and installed the headless proxy as given in https://docs.zyte.com/smart-proxy-manager/headless.html. Please share the parameters with which the headless proxy is started.


Regards,

Hello,


I have cloned the repo and created the executable file with the help of the make command. Then I ran this command to start the server-  

 

./crawlera-headless-proxy -a <API_KEY> -x profile=desktop -x cookies=disable --debug

 In the config.toml i edited the API key, rest of the fields are set to default only. After running the server I'm running this code snippet. - 

 

from selenium import webdriver
headless_proxy = "localhost:3128"
proxy = {
    "proxyType": "manual",
    "httpProxy": headless_proxy,
    "ftpProxy": headless_proxy,
    "sslProxy": headless_proxy,
    "noProxy": "",
}

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument("--ignore-certificate-errors")
chrome_options.set_capability("proxy", proxy)

chromedriver_path = '/Users/prakhar/Downloads/chromedriver'
driver = webdriver.Chrome(executable_path = chromedriver_path, options=chrome_options)
urlpage = "https://google.com"
driver.get(urlpage)

 



In the response, I'm getting "missing_headless_browser_proxy". 


Hi Team,


I'm waiting for your reply. Please help if you can resolve the issue.

Sorry this issue was introduced after the recent release of Smart proxy manager.


 We have now fixed the issue, however you would need to get the latest version of the headless proxy 1.3.1 (https://github.com/zytedata/zyte-smartproxy-headless-proxy/tree/1.3.1)


Please let me know if you still face any issues.


It seems there might be something wrong with the account set up? cause when I run


curl -vx https://proxy.zyte.com:8014 -U <<KEY>>: --cacert zyte-smartproxy-ca.crt https://httpbin.org/ip


I get the result below.


 

* Trying 64.58.117.175...

* TCP_NODELAY set

* Connected to proxy.zyte.com (64.58.117.175) port 8014 (#0)

* ALPN, offering http/1.1

* successfully set certificate verify locations:

* CAfile: /etc/ssl/cert.pem

CApath: none

* TLSv1.2 (OUT), TLS handshake, Client hello (1):

* TLSv1.2 (IN), TLS handshake, Server hello (2):

* TLSv1.2 (IN), TLS handshake, Certificate (11):

* TLSv1.2 (IN), TLS handshake, Server key exchange (12):

* TLSv1.2 (IN), TLS handshake, Server finished (14):

* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

* TLSv1.2 (OUT), TLS handshake, Finished (20):

* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):

* TLSv1.2 (IN), TLS handshake, Finished (20):

* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

* ALPN, server did not agree to a protocol

* Proxy certificate:

* subject: OU=Domain Control Validated; CN=*.zyte.com

* start date: Jan 21 21:36:16 2021 GMT

* expire date: Feb 22 20:06:32 2022 GMT

* subjectAltName: host "proxy.zyte.com" matched cert's "*.zyte.com"

* issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2

* SSL certificate verify ok.

* allocate connect buffer!

* Establish HTTP proxy tunnel to httpbin.org:443

* Proxy auth using Basic with user '<<KEY>>'

> CONNECT httpbin.org:443 HTTP/1.1

> Host: httpbin.org:443

> Proxy-Authorization: Basic ZTkzYWM1YWYyNjQxNDY2OGI2YzJjNWI1YmMyYjk2OGQ6

> User-Agent: curl/7.64.1

> Proxy-Connection: Keep-Alive

>

< HTTP/1.1 200 OK

<

* Proxy replied 200 to CONNECT request

* CONNECT phase completed!

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

* CAfile: zyte-smartproxy-ca.crt

CApath: none

* TLSv1.2 (OUT), TLS handshake, Client hello (1):

* CONNECT phase completed!

* CONNECT phase completed!

* TLSv1.2 (IN), TLS handshake, Server hello (2):

* TLSv1.2 (IN), TLS handshake, Certificate (11):

* TLSv1.2 (IN), TLS handshake, Server key exchange (12):

* TLSv1.2 (IN), TLS handshake, Server finished (14):

* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

* TLSv1.2 (OUT), TLS handshake, Finished (20):

* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):

* TLSv1.2 (IN), TLS handshake, Finished (20):

* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

* ALPN, server did not agree to a protocol

* Server certificate:

* subject: [NONE]

* start date: Sep 6 00:00:00 2021 GMT

* expire date: Dec 10 00:00:00 2023 GMT

* subjectAltName: host "httpbin.org" matched cert's "httpbin.org"

* issuer: C=IE; ST=Munster; L=Cork; O=ScrapingHub; OU=Leading Technology and Professional Services; CN=Crawlera CA; emailAddress=support@scrapinghub.com

* SSL certificate verify ok.

> GET /ip HTTP/1.1

> Host: httpbin.org

> User-Agent: curl/7.64.1

> Accept: */*

>

< HTTP/1.1 400 Bad Request

< X-Crawlera-Error: missing_headless_browser_proxy

< X-Crawlera-Version: 1.51.0

< Content-Length: 30

< Date: Wed, 08 Sep 2021 19:13:30 GMT

* HTTP/1.1 proxy connection set close!

< Proxy-Connection: close

< Connection: close

<

* Closing connection 0

* TLSv1.2 (OUT), TLS alert, close notify (256):

* TLSv1.2 (OUT), TLS alert, close notify (256):

missing_headless_browser_proxy%

This error means that you have setup SPM account with headless but using without https://github.com/zytedata/zyte-smartproxy-headless-proxy

Login to post a comment