Environment variable in scrapinghub not detected

Posted about 4 years ago by Ane

Post a topic

Un Answered

Ane

I want to read an API as an environment variable in ScrapingHub.

I have saved it in Spiders > Settings (in the screenshot), and then I have tried to extract it in Python:

api = os.environ.get('OWM_API')

But when I deploy it to the scrapy cloud, the variable api is None. It doesn't detect the environment variable.

In my scrapinghub.yml I only have the project name and the pointer to the requirements file. What am I doing wrong?

Attachments (1)

scrapinghub.png
22.8 KB

1 Votes

2 Comments

arukomp posted about 4 years ago

I believe the "Raw Settings" panel is not for environment variables, but for Spider settings, similar to what you can find in "settings.py" file.

I do something similar, and here's how it works for me:

1. Add a new line to "settings.py" like so:

OWM_API = os.environ.get('OWM_API')

2. Wherever you need to get this setting, you can fetch it from the spider's 'settings' attribute:

your_token = spider.settings['OWM_API']

## or this one, if you're within the Spider class:
your_token = self.settings['OWN_API']

Hope this helps!

1 Votes

Robb Shecter posted over 2 years ago

I don't understand — `os.environ.get()` fails in settings.py. ?

0 Votes