Start a new topic

Crawlera responding differently Ubuntu server/Mac

Hello Team,

when I run below code from my Mac I get the correct response via Crawlera.

When I run the same code from an Ubuntu/debian server on AWS/Google Cloud '' sends back a 'Are you human?' response via Crawlera.

It is exactly the same code on all machines and 'requests' is the same version on all as well.

I am trying to move a bigger crawler to AWS but this problem with requests is blocking me.

Thanks a lot

import requests

headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36'}

response = requests.get(



        "http": "",





print(response.text) serves https and in your proxies parameter you have only defined "http" so your requests are not going through Crawlera.

You should have:

        "https": "",

1 person likes this

Thanks Nestor! That worked

Login to post a comment