Learn all about the latest trends and best practices in data extraction - Join us at Extract SummitGet tickets
Start a new topic

Job landing page

 We are crawling crowd venues with many seats and our items contain many nested lists. When even one item is rendered in Scrapy Cloud it puts a burden on the browser but when the ten (10) items are rendered on the jobs page my Firefox is spinning for so long that my little i7 4-core starts getting warm and I just end up writing a quick Hubstorage script.


Suggestion: limit render sizes with only showing the first few entries of iterables by default; and, I guess that means drill into the items recursively, hm. Anyway, that would great I think.

Login to post a comment