Python code works on my computer doesn't work with AWS Lambda
V
Victor
started a topic
10 months ago
Hi,
I have Python web scraping code works well on my local machine (Ubuntu 20.04). But when I try to run it on AWS Lambda, I always get Proxy connection error:
[ERROR] ProxyError: HTTPSConnectionPool(host='www.indeed.com', port=443): Max retries exceeded with url: /jobs?q=Software+Engineer&sort=date (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d4156d3d0>: Failed to establish a new connection: [Errno 110] Connection timed out')))
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 16, in lambda_handler
indeed.scraping_job_stats()
File "/var/task/indeed_job_scraper.py", line 945, in scraping_job_stats
scraper.process_job_title_stats(jobTitle)
File "/var/task/indeed_job_scraper.py", line 607, in process_job_title_stats
response = self.proxyAdapter.getURL(url)
File "/var/task/indeed_job_scraper.py", line 115, in getURL
response = requests.get(
File "/opt/python/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/opt/python/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/python/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/opt/python/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/opt/python/requests/adapters.py", line 559, in send
raise ProxyError(e, request=request)END RequestId: 1ebf64b9-3f2e-457f-8e76-63bfde9d69c1
REPORT RequestId: 1ebf64b9-3f2e-457f-8e76-63bfde9d69c1 Duration: 131163.76 ms Billed Duration: 131164 ms Memory Size: 128 MB Max Memory Used: 68 MB Init Duration: 922.33 ms
Request ID
1ebf64b9-3f2e-457f-8e76-63bfde9d69c1
I believe I setup the API token and zyte certificate properly. What could be the issues there?
Thanks
1 Comment
V
Victor
said
10 months ago
The only thing different between my machine and AWS lambda is that I'm using python 3.10 and there is only python 3.9 available on AWS Lambda.
Victor
Hi,
I have Python web scraping code works well on my local machine (Ubuntu 20.04). But when I try to run it on AWS Lambda, I always get Proxy connection error:
I believe I setup the API token and zyte certificate properly. What could be the issues there?
Thanks