I have Python web scraping code works well on my local machine (Ubuntu 20.04). But when I try to run it on AWS Lambda, I always get Proxy connection error:
[ERROR] ProxyError: HTTPSConnectionPool(host='www.indeed.com', port=443): Max retries exceeded with url: /jobs?q=Software+Engineer&sort=date (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d4156d3d0>: Failed to establish a new connection: [Errno 110] Connection timed out')))
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 16, in lambda_handler
indeed.scraping_job_stats()
File "/var/task/indeed_job_scraper.py", line 945, in scraping_job_stats
scraper.process_job_title_stats(jobTitle)
File "/var/task/indeed_job_scraper.py", line 607, in process_job_title_stats
response = self.proxyAdapter.getURL(url)
File "/var/task/indeed_job_scraper.py", line 115, in getURL
response = requests.get(
File "/opt/python/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/opt/python/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/python/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/opt/python/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/opt/python/requests/adapters.py", line 559, in send
raise ProxyError(e, request=request)END RequestId: 1ebf64b9-3f2e-457f-8e76-63bfde9d69c1
REPORT RequestId: 1ebf64b9-3f2e-457f-8e76-63bfde9d69c1 Duration: 131163.76 ms Billed Duration: 131164 ms Memory Size: 128 MB Max Memory Used: 68 MB Init Duration: 922.33 ms
Request ID
1ebf64b9-3f2e-457f-8e76-63bfde9d69c1
I believe I setup the API token and zyte certificate properly. What could be the issues there?
Thanks
0 Votes
1 Comments
V
Victorposted
almost 2 years ago
The only thing different between my machine and AWS lambda is that I'm using python 3.10 and there is only python 3.9 available on AWS Lambda.
Hi,
I have Python web scraping code works well on my local machine (Ubuntu 20.04). But when I try to run it on AWS Lambda, I always get Proxy connection error:
I believe I setup the API token and zyte certificate properly. What could be the issues there?
Thanks
0 Votes
1 Comments
Victor posted almost 2 years ago
The only thing different between my machine and AWS lambda is that I'm using python 3.10 and there is only python 3.9 available on AWS Lambda.
0 Votes
Login to post a comment