Could not obtain new session even after retry

Posted over 3 years ago by Payments Reevoy

Post a topic

Un Answered

Payments Reevoy

I have set up the Zyte Smart Proxy Manager and also downloaded the certitficate from "https://docs.zyte.com/_static/zyte-smartproxy-ca.crt" URL and added to my key chain, marked it as trusted but whenever i'm trying to access any URL from selenium chromedriver. It is not working and giving me invalid certificate error.

Also I'm getting this warning into my logs -

WARN[0136] Could not obtain new session even after retry

Please let me know if there is any other URL for the certificate? Or i've missed something?

0 Votes

8 Comments

Miguel Ortega Salas posted over 1 year ago

Hi,
Same issue is here. It's occurring randomly. Any permanent solution for it?

0 Votes

Renan Castro posted over 2 years ago

I'm having the same issue with Selenium and Chrome.

Have you found any solution??

1 Votes

aurish_hammad_hafeez posted over 3 years ago Admin

This error means that you have setup SPM account with headless but using without https://github.com/zytedata/zyte-smartproxy-headless-proxy

0 Votes

Mikhael Farah posted over 3 years ago

It seems there might be something wrong with the account set up? cause when I run

curl -vx https://proxy.zyte.com:8014 -U <<KEY>>: --cacert zyte-smartproxy-ca.crt https://httpbin.org/ip

I get the result below.

* Trying 64.58.117.175...

* TCP_NODELAY set

* Connected to proxy.zyte.com (64.58.117.175) port 8014 (#0)

* ALPN, offering http/1.1

* successfully set certificate verify locations:

* CAfile: /etc/ssl/cert.pem

CApath: none

* TLSv1.2 (OUT), TLS handshake, Client hello (1):

* TLSv1.2 (IN), TLS handshake, Server hello (2):

* TLSv1.2 (IN), TLS handshake, Certificate (11):

* TLSv1.2 (IN), TLS handshake, Server key exchange (12):

* TLSv1.2 (IN), TLS handshake, Server finished (14):

* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

* TLSv1.2 (OUT), TLS handshake, Finished (20):

* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):

* TLSv1.2 (IN), TLS handshake, Finished (20):

* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

* ALPN, server did not agree to a protocol

* Proxy certificate:

* subject: OU=Domain Control Validated; CN=*.zyte.com

* start date: Jan 21 21:36:16 2021 GMT

* expire date: Feb 22 20:06:32 2022 GMT

* subjectAltName: host "proxy.zyte.com" matched cert's "*.zyte.com"

* issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2

* SSL certificate verify ok.

* allocate connect buffer!

* Establish HTTP proxy tunnel to httpbin.org:443

* Proxy auth using Basic with user '<<KEY>>'

> CONNECT httpbin.org:443 HTTP/1.1

> Host: httpbin.org:443

> Proxy-Authorization: Basic ZTkzYWM1YWYyNjQxNDY2OGI2YzJjNWI1YmMyYjk2OGQ6

> User-Agent: curl/7.64.1

> Proxy-Connection: Keep-Alive

< HTTP/1.1 200 OK

* Proxy replied 200 to CONNECT request

* CONNECT phase completed!

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

* CAfile: zyte-smartproxy-ca.crt

CApath: none

* TLSv1.2 (OUT), TLS handshake, Client hello (1):

* CONNECT phase completed!

* TLSv1.2 (IN), TLS handshake, Server hello (2):

* TLSv1.2 (IN), TLS handshake, Certificate (11):

* TLSv1.2 (IN), TLS handshake, Server key exchange (12):

* TLSv1.2 (IN), TLS handshake, Server finished (14):

* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):

* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):

* TLSv1.2 (OUT), TLS handshake, Finished (20):

* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):

* TLSv1.2 (IN), TLS handshake, Finished (20):

* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

* ALPN, server did not agree to a protocol

* Server certificate:

* subject: [NONE]

* start date: Sep 6 00:00:00 2021 GMT

* expire date: Dec 10 00:00:00 2023 GMT

* subjectAltName: host "httpbin.org" matched cert's "httpbin.org"

* issuer: C=IE; ST=Munster; L=Cork; O=ScrapingHub; OU=Leading Technology and Professional Services; CN=Crawlera CA; [email protected]

* SSL certificate verify ok.

> GET /ip HTTP/1.1

> Host: httpbin.org

> User-Agent: curl/7.64.1

> Accept: */*

< HTTP/1.1 400 Bad Request

< X-Crawlera-Error: missing_headless_browser_proxy

< X-Crawlera-Version: 1.51.0

< Content-Length: 30

< Date: Wed, 08 Sep 2021 19:13:30 GMT

* HTTP/1.1 proxy connection set close!

< Proxy-Connection: close

< Connection: close

* Closing connection 0

* TLSv1.2 (OUT), TLS alert, close notify (256):

missing_headless_browser_proxy%

0 Votes

thriveni posted over 3 years ago Admin

Sorry this issue was introduced after the recent release of Smart proxy manager.

We have now fixed the issue, however you would need to get the latest version of the headless proxy 1.3.1 (https://github.com/zytedata/zyte-smartproxy-headless-proxy/tree/1.3.1)

Please let me know if you still face any issues.

0 Votes

Payments Reevoy posted over 3 years ago

Hi Team,

I'm waiting for your reply. Please help if you can resolve the issue.

0 Votes

Payments Reevoy posted over 3 years ago

Hello,

I have cloned the repo and created the executable file with the help of the make command. Then I ran this command to start the server-

./crawlera-headless-proxy -a <API_KEY> -x profile=desktop -x cookies=disable --debug

In the config.toml i edited the API key, rest of the fields are set to default only. After running the server I'm running this code snippet. -

from selenium import webdriver
headless_proxy = "localhost:3128"
proxy = {
    "proxyType": "manual",
    "httpProxy": headless_proxy,
    "ftpProxy": headless_proxy,
    "sslProxy": headless_proxy,
    "noProxy": "",
}

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument("--ignore-certificate-errors")
chrome_options.set_capability("proxy", proxy)

chromedriver_path = '/Users/prakhar/Downloads/chromedriver'
driver = webdriver.Chrome(executable_path = chromedriver_path, options=chrome_options)
urlpage = "https://google.com"
driver.get(urlpage)

In the response, I'm getting "missing_headless_browser_proxy".

0 Votes

thriveni posted over 3 years ago Admin

Hello,

Please ensure that you have downloaded and installed the headless proxy as given in https://docs.zyte.com/smart-proxy-manager/headless.html. Please share the parameters with which the headless proxy is started.

Regards,

0 Votes