Start a new topic

Use Crawlera with Selenium issues


I followed the instructions to implement Crawlera with Selenium.

When using Chrome, there is a warning NET::ERR_CERT_AUTHORITY_INVALID.

When using Firefox, it goes to the url with a security exception but the page loads very slowly and it times out. 

My code as below

headless_proxy = "localhost:3128"



proxy = Proxy({

    'proxyType': ProxyType.MANUAL,

    'httpProxy': headless_proxy,

    'ftpProxy' : headless_proxy,

    'sslProxy' : headless_proxy,

    'noProxy' : ''



chrome_options = Options()



capabilities = dict(DesiredCapabilities.CHROME)


driver = webdriver.Chrome(desired_capabilities=capabilities, executable_path='chromedriver', options=chrome_options)


# Firefox

firefox_capabilities = webdriver.DesiredCapabilities.FIREFOX

firefox_capabilities['marionette'] = True


firefox_capabilities['proxy'] = {

    "proxyType": "MANUAL",

    "httpProxy": headless_proxy,

    "ftpProxy": headless_proxy,

    "sslProxy": headless_proxy



driver = webdriver.Firefox(capabilities=firefox_capabilities)


I am using scrapy with crawlera and also tried Splash + scrapy-splash but there is a warning " scrapy-splash Call to deprecated function to_native_str. Use to_unicode instead". I followed this instructions

I just need to find a solution that works for scrapy+crawlera+rendering or crawelra+selenium. 

1 person has this question
Login to post a comment