videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic
Answered

Selenium support

 Hi,

I need to use selenium in my spider to work. Is it even possible? If yes, how to do it?


Best Answer

This is only possible by deploying a Docker custom image https://shub.readthedocs.io/en/stable/deploy-custom-image.html#deploy-custom-image, note that this feature is only available to users subscribed to at least 1 Scrapy Cloud unit.


Hi, mind if i ask whether using a docker custom image worked or not? My spider needs to use Selenium for firefox too and this looks like would solve my problem.

Answer

This is only possible by deploying a Docker custom image https://shub.readthedocs.io/en/stable/deploy-custom-image.html#deploy-custom-image, note that this feature is only available to users subscribed to at least 1 Scrapy Cloud unit.

Could I be shown an example docker file to integrate selenium into scrapy unit? How do I unzip the chromedriver?

You need to either create an image of your own or pull someone else's base image. I'm attaching the one I used. The base is Ubuntu and it has python2 and both chromedriver and geckodriver installed. Remember that deploying docker images is a paid feature so you need to buy at least 1 unit.

(277 Bytes)

Thanks a lot for sharing. I figured it out but appreciate!

Login to post a comment