videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic
Answered

Scraping issue

 Hi, I am trying to access website carmax.com to scrape some vehicle details and was going to use Splash for it, so i got open source version , installed it and try to get data, same as curl request i am getting on bot protection even with Splash. So I was wondering if your hosted version are more advanced so I can get access to that site with it ?


Best Answer

Hosted versions are exactly the same. You should consider using a proxy service like Crawlera if you're having issues with bot protection. You can implement Crawlera in your Splash using a Lua script, see: https://support.scrapinghub.com/support/solutions/articles/22000188428-using-crawlera-with-splash-scrapy and https://support.scrapinghub.com/support/solutions/articles/22000203566-using-crawlera-with-splash-python-requests-library.

1 Comment

Answer

Hosted versions are exactly the same. You should consider using a proxy service like Crawlera if you're having issues with bot protection. You can implement Crawlera in your Splash using a Lua script, see: https://support.scrapinghub.com/support/solutions/articles/22000188428-using-crawlera-with-splash-scrapy and https://support.scrapinghub.com/support/solutions/articles/22000203566-using-crawlera-with-splash-python-requests-library.

Login to post a comment