Improved Headless Browser experience on Smart Proxy Manager
Posted
over 3 years ago
by Pablo Hoffman
Pablo HoffmanAdmin
Smart Proxy Manager (formerly, Crawlera) was designed for Scrapy, and browser-less environments where only the raw HTML is fetched, but the evolving nature of the web towards more reactive, javascript based apps have led scraping professionals to adopt headless browsers such as Puppeteer, Selenium and Playwright as their preferred tool of choice for web scraping javascript-intensive websites.
With this in mind, we've been working to improve the experience using headless browsers with our Smart Proxy Manager, and we will keep working on making it easier to integrate your headless browser solution in the coming months.
Today we are releasing a change in Smart Proxy Manager setup screen. When you set up a new Smart Proxy account (and its corresponding API key) you will now be able to select between two type of integrations
Proxy API integration: the classic Smart Proxy Manager account
Headless Browser integration: which is optimised for usage in headless browsers
Here is how the new Setup screen looks like now, note the new integration type selector:
We also require using our companion tool, the headless browser proxy, to ensure optimal performance and avoid getting billed for unnecessary requests.
You can find the new documentation for integrating headless browsers here with much simpler sample codes for Playwright, Puppeteer, Selenium and Splash.
We hope you enjoy these improvements, let us know in the comments what else you would like to see improved.
Happy scraping!
2 Votes
2 Comments
Sorted by
s
sabri sabriposted
almost 2 years ago
suspended
WHY MY ACCOUNT IS suspended
0 Votes
R
R Cposted
about 3 years ago
Great info! Understand that there is a companion tool required for the improvements. Just want to know if this can also be run in google colab as well. If yes, appreciate if more details about how this can be done here please. Thanks.
Smart Proxy Manager (formerly, Crawlera) was designed for Scrapy, and browser-less environments where only the raw HTML is fetched, but the evolving nature of the web towards more reactive, javascript based apps have led scraping professionals to adopt headless browsers such as Puppeteer, Selenium and Playwright as their preferred tool of choice for web scraping javascript-intensive websites.
With this in mind, we've been working to improve the experience using headless browsers with our Smart Proxy Manager, and we will keep working on making it easier to integrate your headless browser solution in the coming months.
Today we are releasing a change in Smart Proxy Manager setup screen. When you set up a new Smart Proxy account (and its corresponding API key) you will now be able to select between two type of integrations
Here is how the new Setup screen looks like now, note the new integration type selector:
We also require using our companion tool, the headless browser proxy, to ensure optimal performance and avoid getting billed for unnecessary requests.
You can find the new documentation for integrating headless browsers here with much simpler sample codes for Playwright, Puppeteer, Selenium and Splash.
We hope you enjoy these improvements, let us know in the comments what else you would like to see improved.
Happy scraping!
2 Votes
2 Comments
sabri sabri posted almost 2 years ago
suspended
WHY MY ACCOUNT IS suspended
0 Votes
R C posted about 3 years ago
Great info! Understand that there is a companion tool required for the improvements. Just want to know if this can also be run in google colab as well. If yes, appreciate if more details about how this can be done here please. Thanks.
0 Votes
Login to post a comment