videocamWeb Data Extraction Summit - September 30th, 2021.
Join some of the greatest minds in web scraping to educate, inspire, and innovate.
Register for free!
Start a new topic

boto3 - botocore.exceptions.NoCredentialsError: Unable to locate credentials

Hey all,


I want to achieve the following:

Invoking an AWS Lambda function.


Situation:

- I can invoke the Lambda locally

- I'm using a ~/.aws/credentials file, where I store the AWS access and secret key


Challenge:

- I set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the Spider Settings UI

- However, when running the spider on Zyte, I get the following error: botocore.exceptions.NoCredentialsError: Unable to locate credentials


When I use

- os.environ["AWS_ACCESS_KEY_ID"] = <my_key>

- os.environ["AWS_SECRET_ACCESS_KEY"] = <my_key>

in the spider itself it works fine. But it is really against all best-practices to store the keys in the code.


Does anyone know why the environment variables are not picked up from the Spider Settings?



Best regards


1 person has this question
1 Comment

I solved it now by reading the following:

shub_settings = json.loads(os.environ['SHUB_SETTINGS'])

aws_access_key_id = shub_settings['project_settings']['AWS_ACCESS_KEY_ID']

Login to post a comment