Zyte Support Center

Hi,

I have Python web scraping code works well on my local machine (Ubuntu 20.04). But when I try to run it on AWS Lambda, I always get Proxy connection error:

[ERROR] ProxyError: HTTPSConnectionPool(host='www.indeed.com', port=443): Max retries exceeded with url: /jobs?q=Software+Engineer&sort=date (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d4156d3d0>: Failed to establish a new connection: [Errno 110] Connection timed out')))
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 16, in lambda_handler
    indeed.scraping_job_stats()
  File "/var/task/indeed_job_scraper.py", line 945, in scraping_job_stats
    scraper.process_job_title_stats(jobTitle)
  File "/var/task/indeed_job_scraper.py", line 607, in process_job_title_stats
    response = self.proxyAdapter.getURL(url)
  File "/var/task/indeed_job_scraper.py", line 115, in getURL
    response = requests.get(
  File "/opt/python/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/opt/python/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/python/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/python/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/opt/python/requests/adapters.py", line 559, in send
    raise ProxyError(e, request=request)END RequestId: 1ebf64b9-3f2e-457f-8e76-63bfde9d69c1
REPORT RequestId: 1ebf64b9-3f2e-457f-8e76-63bfde9d69c1	Duration: 131163.76 ms	Billed Duration: 131164 ms	Memory Size: 128 MB	Max Memory Used: 68 MB	Init Duration: 922.33 ms

Request ID
1ebf64b9-3f2e-457f-8e76-63bfde9d69c1

 I believe I setup the API token and zyte certificate properly. What could be the issues there?

Thanks  

The only thing different between my machine and AWS lambda is that I'm using python 3.10 and there is only python 3.9 available on AWS Lambda.