videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic

Using Zyte with Selenium?

I am using this Ruby scrip to test Zyte with Selenium:


(Selenium Version: 4.0.0.beta2)

 

   

proxy = Selenium::WebDriver::Proxy.new(http: "API_KEY:@proxy.crawlera.com:8011")
cap = Selenium::WebDriver::Remote::Capabilities.chrome(proxy: proxy)
driver = Selenium::WebDriver.for(:chrome, capabilities: cap)
driver.navigate.to("http://httpbin.org/ip")
puts "content: #{driver.page_source}"
driver.close

  

 But it is returning my ip instead of the proxy IP


If I use another proxy (without authentication) I get the proper proxy IP:


 

proxy = Selenium::WebDriver::Proxy.new(http: "185.235.41.85:80")
cap = Selenium::WebDriver::Remote::Capabilities.chrome(proxy: proxy)
driver = Selenium::WebDriver.for(:chrome, capabilities: cap)
driver.navigate.to("http://httpbin.org/ip")
puts "content: #{driver.page_source}"
driver.close

 Result: 

  

{
  "origin": "185.235.41.85"
} 

  

1 Comment

The only way I found to make it possible is using the crawlera-headless-proxy:


https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium


Here is some example configuration if you have problems with ssl:


https://support.zyte.com/support/discussions/topics/22000014124?page=1

Login to post a comment