Hello, I encountered a problem while trying to use Selenium's PhantomJS web driver with specified proxy - crawlera. There is an article about polipo with selenium, but it won't work and anyway, most of the time I work with Windows, so polipo will be problematic.
I tried to do it like:
service_args = [ '--proxy="proxy.crawlera.com:8010"', '--proxy-auth="apikeyweirdcode:"' ]
self.driver = webdriver.PhantomJS(service_args=service_args)
But it won't work, tried with colon and without, with double quotes, without and always when I scrape whatsmyip website with Selenium (by Selenium Middleware, I get a page then in middleware I get specific div with IP address and then print it) I get my own IP.
After this Selenium printing, I get to scrapy's shell and here when I take the same div with IP, I get some Crawlera's IP, not mine.
Did anyone use selenium with crawlera and it worked?
Cheers.
Best Answer
v
vaz
said
about 6 years ago
Hey TeamScrapy, we have provided useful documentation to use Crawlera with different languages and browsers, please check our section: Crawlera Examples
If you still having issues to connect Crawlera, please consider to Hire our experts to provide coding assistance, set or even deploy the crawlers for you. If interested let us know through: https://scrapinghub.com/quote
Hey TeamScrapy, we have provided useful documentation to use Crawlera with different languages and browsers, please check our section: Crawlera Examples
If you still having issues to connect Crawlera, please consider to Hire our experts to provide coding assistance, set or even deploy the crawlers for you. If interested let us know through: https://scrapinghub.com/quote
Best regards,
Pablo
d
delivery
said
about 5 years ago
Hi Pablo, I'll be wondered if you can point us to the right solution of this problem.
teamscrappy
Hello, I encountered a problem while trying to use Selenium's PhantomJS web driver with specified proxy - crawlera. There is an article about polipo with selenium, but it won't work and anyway, most of the time I work with Windows, so polipo will be problematic.
I tried to do it like:
Hey TeamScrapy, we have provided useful documentation to use Crawlera with different languages and browsers, please check our section: Crawlera Examples
If you still having issues to connect Crawlera, please consider to Hire our experts to provide coding assistance, set or even deploy the crawlers for you. If interested let us know through: https://scrapinghub.com/quote
Best regards,
Pablo
- Oldest First
- Popular
- Newest First
Sorted by Oldest Firstvaz
Hey TeamScrapy, we have provided useful documentation to use Crawlera with different languages and browsers, please check our section: Crawlera Examples
If you still having issues to connect Crawlera, please consider to Hire our experts to provide coding assistance, set or even deploy the crawlers for you. If interested let us know through: https://scrapinghub.com/quote
Best regards,
Pablo
delivery
Hi Pablo, I'll be wondered if you can point us to the right solution of this problem.
Evgeny.
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
See all 381 topics