Some website will response different content in different IP. So I want to know the difference from my debug computer. How can I know about the response in cloud?
Best Answer
n
nestor
said
over 5 years ago
If it's less than 1MB you can store the full HTML as an item, if larger than 1MB you'll need an external storage like S3.
winterbesos
Some website will response different content in different IP. So I want to know the difference from my debug computer. How can I know about the response in cloud?
If it's less than 1MB you can store the full HTML as an item, if larger than 1MB you'll need an external storage like S3.
Otherwise, you can use the pagestorage addon: https://support.scrapinghub.com/support/solutions/articles/22000200403-page-storage-addon
nestor
If it's less than 1MB you can store the full HTML as an item, if larger than 1MB you'll need an external storage like S3.
Otherwise, you can use the pagestorage addon: https://support.scrapinghub.com/support/solutions/articles/22000200403-page-storage-addon
-
Crawlera 503 Ban
-
Amazon scraping speed
-
Website redirects
-
Error Code 429 Too Many Requests
-
Bing
-
Subscribed to Crawlera but saying Not Subscribed
-
Selenium with c#
-
Using Crawlera with browsermob
-
CRAWLERA_PRESERVE_DELAY leads to error
-
How to connect Selenium PhantomJS to Crawlera?
See all 399 topics