I want to achieve the following:
Invoking an AWS Lambda function.
- I can invoke the Lambda locally
- I'm using a ~/.aws/credentials file, where I store the AWS access and secret key
- I set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the Spider Settings UI
- However, when running the spider on Zyte, I get the following error: botocore.exceptions.NoCredentialsError: Unable to locate credentials
When I use
- os.environ["AWS_ACCESS_KEY_ID"] = <my_key>
- os.environ["AWS_SECRET_ACCESS_KEY"] = <my_key>
in the spider itself it works fine. But it is really against all best-practices to store the keys in the code.
Does anyone know why the environment variables are not picked up from the Spider Settings?
I solved it now by reading the following:
shub_settings = json.loads(os.environ['SHUB_SETTINGS'])
aws_access_key_id = shub_settings['project_settings']['AWS_ACCESS_KEY_ID']