How to connect Selenium PhantomJS to Crawlera?

Posted almost 8 years ago by teamscrappy

Post a topic

Answered

teamscrappy

Hello, I encountered a problem while trying to use Selenium's PhantomJS web driver with specified proxy - crawlera. There is an article about polipo with selenium, but it won't work and anyway, most of the time I work with Windows, so polipo will be problematic.

I tried to do it like:

service_args = [
    '--proxy="proxy.crawlera.com:8010"',
    '--proxy-auth="apikeyweirdcode:"'
]




self.driver = webdriver.PhantomJS(service_args=service_args)

But it won't work, tried with colon and without, with double quotes, without and always when I scrape whatsmyip website with Selenium (by Selenium Middleware, I get a page then in middleware I get specific div with IP address and then print it) I get my own IP. 
After this Selenium printing, I get to scrapy's shell and here when I take the same div with IP, I get some Crawlera's IP, not mine. 

Did anyone use selenium with crawlera and it worked? 


Cheers.

0 Votes

vaz posted over 7 years ago Best Answer

Hey TeamScrapy, we have provided useful documentation to use Crawlera with different languages and browsers, please check our section: Crawlera Examples

If you still having issues to connect Crawlera, please consider to Hire our experts to provide coding assistance, set or even deploy the crawlers for you. If interested let us know through: https://scrapinghub.com/quote

Best regards,

Pablo

0 Votes

2 Comments

delivery posted almost 7 years ago

Hi Pablo, I'll be wondered if you can point us to the right solution of this problem.

Evgeny.

0 Votes

vaz posted over 7 years ago Answer

Hey TeamScrapy, we have provided useful documentation to use Crawlera with different languages and browsers, please check our section: Crawlera Examples

Best regards,

Pablo

0 Votes