Python code works on my computer doesn't work with AWS Lambda

Posted almost 2 years ago by Victor

Post a topic
Un Answered
V
Victor

Hi,


I have Python web scraping code works well on my local machine (Ubuntu 20.04). But when I try to run it on AWS Lambda, I always get Proxy connection error:

 

[ERROR] ProxyError: HTTPSConnectionPool(host='www.indeed.com', port=443): Max retries exceeded with url: /jobs?q=Software+Engineer&sort=date (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d4156d3d0>: Failed to establish a new connection: [Errno 110] Connection timed out')))
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 16, in lambda_handler
    indeed.scraping_job_stats()
  File "/var/task/indeed_job_scraper.py", line 945, in scraping_job_stats
    scraper.process_job_title_stats(jobTitle)
  File "/var/task/indeed_job_scraper.py", line 607, in process_job_title_stats
    response = self.proxyAdapter.getURL(url)
  File "/var/task/indeed_job_scraper.py", line 115, in getURL
    response = requests.get(
  File "/opt/python/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/opt/python/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/python/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/python/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/opt/python/requests/adapters.py", line 559, in send
    raise ProxyError(e, request=request)END RequestId: 1ebf64b9-3f2e-457f-8e76-63bfde9d69c1
REPORT RequestId: 1ebf64b9-3f2e-457f-8e76-63bfde9d69c1	Duration: 131163.76 ms	Billed Duration: 131164 ms	Memory Size: 128 MB	Max Memory Used: 68 MB	Init Duration: 922.33 ms

Request ID
1ebf64b9-3f2e-457f-8e76-63bfde9d69c1

 I believe I setup the API token and zyte certificate properly. What could be the issues there?

Thanks  

0 Votes


1 Comments

V

Victor posted almost 2 years ago

The only thing different between my machine and AWS lambda is that I'm using python 3.10 and there is only python 3.9 available on AWS Lambda.

0 Votes

Login to post a comment