Crawlera very slow (Puppeteer & Pages Jaunes)

Posted over 6 years ago by jeromearsene2

Post a topic
Answered
j
jeromearsene2

I use crawlera with Puppeteer for https://www.pagesjaunes.fr

My code is similar to the official example: https://support.scrapinghub.com/support/solutions/articles/22000220800-using-crawlera-with-puppeteer


I have test with multiple possibility. I have this result for load the page:

Crawlera Proxy

 test 1: 90294.232ms 

 test 2: 129361.872ms 

 test 3: 176996.368ms

VPN

 test 1: 12641.482ms 

 test 2: 5216.013ms

Normal

 test 1: 5418.336ms


Have you an idea ? A reason ?


Thanks !

0 Votes

nestor

nestor posted over 6 years ago Admin Best Answer

This is normal. Crawlera throttles requests by default, it's priority is to try and avoid bans, not response times. We're currently working on better support for headless browsers, but in the meantime, try disabling loading resources through the proxy.

0 Votes


4 Comments

Sorted by
Tomas Rinke

Tomas Rinke posted over 5 years ago

0 Votes

H

Harshvardhan Gupta posted almost 6 years ago

Has there been any update on this? I am also facing the same issue with headless chrome.

0 Votes

j

jeromearsene2 posted over 6 years ago

Thanks for your response Nestor. 

I desactivate the headless mode for all my examples. 


Can you tell me more for "try disabling loading resources through the proxy" ? How do you do this ?


Thanks

0 Votes

nestor

nestor posted over 6 years ago Admin Answer

This is normal. Crawlera throttles requests by default, it's priority is to try and avoid bans, not response times. We're currently working on better support for headless browsers, but in the meantime, try disabling loading resources through the proxy.

0 Votes

Login to post a comment