Hello, I encountered a problem while trying to use Selenium's PhantomJS web driver with specified proxy - crawlera. There is an article about polipo with selenium, but it won't work and anyway, most of the time I work with Windows, so polipo will be problematic.
I tried to do it like:
service_args = [ '--proxy="proxy.crawlera.com:8010"', '--proxy-auth="apikeyweirdcode:"' ]
self.driver = webdriver.PhantomJS(service_args=service_args)
But it won't work, tried with colon and without, with double quotes, without and always when I scrape whatsmyip website with Selenium (by Selenium Middleware, I get a page then in middleware I get specific div with IP address and then print it) I get my own IP.
After this Selenium printing, I get to scrapy's shell and here when I take the same div with IP, I get some Crawlera's IP, not mine.
Did anyone use selenium with crawlera and it worked?
Cheers.
0 Votes
vaz posted
about 7 years ago
Best Answer
Hey TeamScrapy, we have provided useful documentation to use Crawlera with different languages and browsers, please check our section: Crawlera Examples
If you still having issues to connect Crawlera, please consider to Hire our experts to provide coding assistance, set or even deploy the crawlers for you. If interested let us know through: https://scrapinghub.com/quote
Best regards,
Pablo
0 Votes
2 Comments
Sorted by
d
deliveryposted
over 6 years ago
Hi Pablo, I'll be wondered if you can point us to the right solution of this problem.
Evgeny.
0 Votes
vazposted
about 7 years ago
Answer
Hey TeamScrapy, we have provided useful documentation to use Crawlera with different languages and browsers, please check our section: Crawlera Examples
If you still having issues to connect Crawlera, please consider to Hire our experts to provide coding assistance, set or even deploy the crawlers for you. If interested let us know through: https://scrapinghub.com/quote
Hello, I encountered a problem while trying to use Selenium's PhantomJS web driver with specified proxy - crawlera. There is an article about polipo with selenium, but it won't work and anyway, most of the time I work with Windows, so polipo will be problematic.
I tried to do it like:
0 Votes
vaz posted about 7 years ago Best Answer
Hey TeamScrapy, we have provided useful documentation to use Crawlera with different languages and browsers, please check our section: Crawlera Examples
If you still having issues to connect Crawlera, please consider to Hire our experts to provide coding assistance, set or even deploy the crawlers for you. If interested let us know through: https://scrapinghub.com/quote
Best regards,
Pablo
0 Votes
2 Comments
delivery posted over 6 years ago
Hi Pablo, I'll be wondered if you can point us to the right solution of this problem.
Evgeny.
0 Votes
vaz posted about 7 years ago Answer
Hey TeamScrapy, we have provided useful documentation to use Crawlera with different languages and browsers, please check our section: Crawlera Examples
If you still having issues to connect Crawlera, please consider to Hire our experts to provide coding assistance, set or even deploy the crawlers for you. If interested let us know through: https://scrapinghub.com/quote
Best regards,
Pablo
0 Votes
Login to post a comment