when I run below code from my Mac I get the correct response via Crawlera.
When I run the same code from an Ubuntu/debian server on AWS/Google Cloud 'https://suchen.mobile.de' sends back a 'Are you human?' response via Crawlera.
It is exactly the same code on all machines and 'requests' is the same version on all as well.
I am trying to move a bigger crawler to AWS but this problem with requests is blocking me.
Thanks a lot
import requests
headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36'}
Hello Team,
when I run below code from my Mac I get the correct response via Crawlera.
When I run the same code from an Ubuntu/debian server on AWS/Google Cloud 'https://suchen.mobile.de' sends back a 'Are you human?' response via Crawlera.
It is exactly the same code on all machines and 'requests' is the same version on all as well.
I am trying to move a bigger crawler to AWS but this problem with requests is blocking me.
Thanks a lot
import requests
headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36'}
response = requests.get(
"https://suchen.mobile.de/fahrzeuge/search.html?dam=0&isSearchRequest=true&ms=17200&sfmr=false&vc=Car",
proxies={
"http": "http://API-KEY:@proxy.crawlera.com:8010/",
},
headers=headers,
timeout=600,
)
print(response.text)
0 Votes
2 Comments
nestor posted about 4 years ago Admin
suchen.mobile.de serves https and in your proxies parameter you have only defined "http" so your requests are not going through Crawlera.
You should have:
proxies={ "https": "http://API-KEY:@proxy.crawlera.com:8010/", },1 Votes
Malte Kerssenfischer posted about 4 years ago
Thanks Nestor! That worked
0 Votes
Login to post a comment