I have test with multiple possibility. I have this result for load the page:
Crawlera Proxy:
test 1: 90294.232ms
test 2: 129361.872ms
test 3: 176996.368ms
VPN:
test 1: 12641.482ms
test 2: 5216.013ms
Normal:
test 1: 5418.336ms
Have you an idea ? A reason ?
Thanks !
0 Votes
nestor posted
over 6 years ago
AdminBest Answer
This is normal. Crawlera throttles requests by default, it's priority is to try and avoid bans, not response times. We're currently working on better support for headless browsers, but in the meantime, try disabling loading resources through the proxy.
Has there been any update on this? I am also facing the same issue with headless chrome.
0 Votes
j
jeromearsene2posted
over 6 years ago
Thanks for your response Nestor.
I desactivate the headless mode for all my examples.
Can you tell me more for "try disabling loading resources through the proxy" ? How do you do this ?
Thanks
0 Votes
nestorposted
over 6 years ago
AdminAnswer
This is normal. Crawlera throttles requests by default, it's priority is to try and avoid bans, not response times. We're currently working on better support for headless browsers, but in the meantime, try disabling loading resources through the proxy.
I use crawlera with Puppeteer for https://www.pagesjaunes.fr
My code is similar to the official example: https://support.scrapinghub.com/support/solutions/articles/22000220800-using-crawlera-with-puppeteer
I have test with multiple possibility. I have this result for load the page:
Crawlera Proxy:
test 1: 90294.232ms
test 2: 129361.872ms
test 3: 176996.368ms
VPN:
test 1: 12641.482ms
test 2: 5216.013ms
Normal:
test 1: 5418.336ms
Have you an idea ? A reason ?
Thanks !
0 Votes
nestor posted over 6 years ago Admin Best Answer
This is normal. Crawlera throttles requests by default, it's priority is to try and avoid bans, not response times. We're currently working on better support for headless browsers, but in the meantime, try disabling loading resources through the proxy.
0 Votes
4 Comments
Tomas Rinke posted over 5 years ago
Gupta, what about this option? https://support.scrapinghub.com/support/solutions/articles/22000234854-how-to-use-crawlera-with-headless-browsers
0 Votes
Harshvardhan Gupta posted almost 6 years ago
Has there been any update on this? I am also facing the same issue with headless chrome.
0 Votes
jeromearsene2 posted over 6 years ago
Thanks for your response Nestor.
I desactivate the headless mode for all my examples.
Can you tell me more for "try disabling loading resources through the proxy" ? How do you do this ?
Thanks
0 Votes
nestor posted over 6 years ago Admin Answer
This is normal. Crawlera throttles requests by default, it's priority is to try and avoid bans, not response times. We're currently working on better support for headless browsers, but in the meantime, try disabling loading resources through the proxy.
0 Votes
Login to post a comment