videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic

render_all does not render fullheight for inline containers

I want to render the full height of my page, but if the site contains an inline container it does not work.


It works e.g. with Wikipedia but not my site.
The following two examples describe it:
- https://simon-frey.eu/splashTest/body.html works as expected with render_all=1
- https://simon-frey.eu/splashTest/container.html does not get rendered completely.

The difference is that in the container.html example the body itself has height:100%; and the container inside of it handles the overflow.

Is there a way to make splash render the complete site in that case? One workaround I figured was increasing the height of the virtual browser, but that only works in a few cases and is not reliable.

Hi there!


I'll run some tests and try to reproduce the issue. 

Hi again,


I ran some tests and could not work around this limitation using regular render.png or render.jpg endpoints.


My recommendation is that your try using the execute endpoint. Programatically you will have access to the page elements and you will be able to not only read elements attributes (like sizes), but also change them.

With the hight at hand you can then resize the viewport to fit your needs: https://splash.readthedocs.io/en/stable/scripting-ref.html#splash-set-viewport-size

Thanks for your help so far! Do you have any resource on hand how I could archive to capture the full height of the website with your proposed solution (the execution endpoint)?

Hi!


I can try to make a sample script. I'll update you as soon as I have a POC.

Login to post a comment