Start a new topic

Environment variable in scrapinghub not detected

I want to read an API as an environment variable in ScrapingHub. 


I have saved it in Spiders > Settings (in the screenshot), and then I have tried to extract it in Python:


api = os.environ.get('OWM_API')


But when I deploy it to the scrapy cloud, the variable api is None. It doesn't detect the environment variable.


In my scrapinghub.yml I only have the project name and the pointer to the requirements file. What am I doing wrong?



1 person has this question

I believe the "Raw Settings" panel is not for environment variables, but for Spider settings, similar to what you can find in "settings.py" file.

I do something similar, and here's how it works for me:

1. Add a new line to "settings.py" like so:

OWM_API = os.environ.get('OWM_API')

 

2. Wherever you need to get this setting, you can fetch it from the spider's 'settings' attribute: 

your_token = spider.settings['OWM_API']

## or this one, if you're within the Spider class:
your_token = self.settings['OWN_API']

 

Hope this helps! 


1 person likes this

I don't understand — `os.environ.get()` fails in settings.py. ?

Login to post a comment