Improved Headless Browser experience on Smart Proxy Manager

Smart Proxy Manager (formerly, Crawlera) was designed for Scrapy, and browser-less environments where only the raw HTML is fetched, but the evolving nature of the web towards more reactive, javascript based apps have led scraping professionals to adopt headless browsers such as Puppeteer, Selenium and Playwright as their preferred tool of choice for web scraping javascript-intensive websites.

With this in mind, we've been working to improve the experience using headless browsers with our Smart Proxy Manager, and we will keep working on making it easier to integrate your headless browser solution in the coming months.

Today we are releasing a change in Smart Proxy Manager setup screen. When you set up a new Smart Proxy account (and its corresponding API key) you will now be able to select between two type of integrations

  • Proxy API integration: the classic Smart Proxy Manager account
  • Headless Browser integration: which is optimised for usage in headless browsers

Here is how the new Setup screen looks like now, note the new integration type selector:

We also require using our companion tool, the headless browser proxy, to ensure optimal performance and avoid getting billed for unnecessary requests.

You can find the new documentation for integrating headless browsers here with much simpler sample codes for Playwright, Puppeteer, Selenium and Splash.

We hope you enjoy these improvements, let us know in the comments what else you would like to see improved.

Happy scraping!

2 people like this

Great info! Understand that there is a companion tool required for the improvements. Just want to know if this can also be run in google colab as well. If yes, appreciate if more details about how this can be done here please. Thanks.



Login to post a comment