How can we help you today?

Start a new topic
Discussions Zyte Smart Proxy Manager (formerly Crawlera) Technical Help
Answered

Crawlera very slow (Puppeteer & Pages Jaunes)

I use crawlera with Puppeteer for https://www.pagesjaunes.fr

My code is similar to the official example: https://support.scrapinghub.com/support/solutions/articles/22000220800-using-crawlera-with-puppeteer


I have test with multiple possibility. I have this result for load the page:

Crawlera Proxy: 

 test 1: 90294.232ms 

 test 2: 129361.872ms 

 test 3: 176996.368ms

VPN: 

 test 1: 12641.482ms 

 test 2: 5216.013ms

Normal: 

 test 1: 5418.336ms


Have you an idea ? A reason ?


Thanks !

Best Answer

This is normal. Crawlera throttles requests by default, it's priority is to try and avoid bans, not response times. We're currently working on better support for headless browsers, but in the meantime, try disabling loading resources through the proxy.

4 CommentsSorted by Oldest First
Answer

This is normal. Crawlera throttles requests by default, it's priority is to try and avoid bans, not response times. We're currently working on better support for headless browsers, but in the meantime, try disabling loading resources through the proxy.

Thanks for your response Nestor. 

I desactivate the headless mode for all my examples. 


Can you tell me more for "try disabling loading resources through the proxy" ? How do you do this ?


Thanks

Has there been any update on this? I am also facing the same issue with headless chrome.

Login to post a comment
More topics in Technical Help
See all 401 topics