Crawlera responding differently Ubuntu server/Mac

Posted about 4 years ago by Malte Kerssenfischer

Post a topic
Un Answered
M
Malte Kerssenfischer

Hello Team,


when I run below code from my Mac I get the correct response via Crawlera.


When I run the same code from an Ubuntu/debian server on AWS/Google Cloud 'https://suchen.mobile.de' sends back a 'Are you human?' response via Crawlera.


It is exactly the same code on all machines and 'requests' is the same version on all as well.


I am trying to move a bigger crawler to AWS but this problem with requests is blocking me.


Thanks a lot



import requests


headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36'}


response = requests.get(

    "https://suchen.mobile.de/fahrzeuge/search.html?dam=0&isSearchRequest=true&ms=17200&sfmr=false&vc=Car",

    proxies={

        "http": "http://API-KEY:@proxy.crawlera.com:8010/",

    },

    headers=headers,

    timeout=600,

)

print(response.text)

0 Votes


2 Comments

Sorted by
nestor

nestor posted about 4 years ago Admin

suchen.mobile.de serves https and in your proxies parameter you have only defined "http" so your requests are not going through Crawlera.


You should have:

proxies={
        "https": "http://API-KEY:@proxy.crawlera.com:8010/",
},


1 Votes

M

Malte Kerssenfischer posted about 4 years ago

Thanks Nestor! That worked


0 Votes

Login to post a comment