Learn all about the latest trends and best practices in data extraction - Join us at Extract SummitGet tickets
Start a new topic

Crawlera responding differently Ubuntu server/Mac

Hello Team,


when I run below code from my Mac I get the correct response via Crawlera.


When I run the same code from an Ubuntu/debian server on AWS/Google Cloud 'https://suchen.mobile.de' sends back a 'Are you human?' response via Crawlera.


It is exactly the same code on all machines and 'requests' is the same version on all as well.


I am trying to move a bigger crawler to AWS but this problem with requests is blocking me.


Thanks a lot



import requests


headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36'}


response = requests.get(

    "https://suchen.mobile.de/fahrzeuge/search.html?dam=0&isSearchRequest=true&ms=17200&sfmr=false&vc=Car",

    proxies={

        "http": "http://API-KEY:@proxy.crawlera.com:8010/",

    },

    headers=headers,

    timeout=600,

)

print(response.text)


suchen.mobile.de serves https and in your proxies parameter you have only defined "http" so your requests are not going through Crawlera.


You should have:

proxies={
        "https": "http://API-KEY:@proxy.crawlera.com:8010/",
},



1 person likes this

Thanks Nestor! That worked


Login to post a comment