We are crawling crowd venues with many seats and our items contain many nested lists. When even one item is rendered in Scrapy Cloud it puts a burden on the browser but when the ten (10) items are rendered on the jobs page my Firefox is spinning for so long that my little i7 4-core starts getting warm and I just end up writing a quick Hubstorage script.
Suggestion: limit render sizes with only showing the first few entries of iterables by default; and, I guess that means drill into the items recursively, hm. Anyway, that would great I think.
We are crawling crowd venues with many seats and our items contain many nested lists. When even one item is rendered in Scrapy Cloud it puts a burden on the browser but when the ten (10) items are rendered on the jobs page my Firefox is spinning for so long that my little i7 4-core starts getting warm and I just end up writing a quick Hubstorage script.
Suggestion: limit render sizes with only showing the first few entries of iterables by default; and, I guess that means drill into the items recursively, hm. Anyway, that would great I think.
0 Votes
0 Comments
Login to post a comment