Start a new topic

How to set environment variables on Scrapy cloud

My scrapy project makes use of certain environment variables. I want to migrate my project to Scrapy cloud and need to set environment variable on cloud environment. Can somebody suggest how to achieve the same.


4 people have this question
1 Comment

Hi, 


As suggested by this post, by strict definition, the environment variables (i.e os.environ["FOO"]) is usually not the values you'll want to use for custom external configuration. 


However you should use spider settings or arguments.


Arguments are those config values per execution of a spider, and are available in init() function of your spider, like:


    

class Spider(CrawlSpider):
    name = 'foo'
    def __init__(self, *a, **kw):
        custom_setting=kw.get('some_setting')

    


Spider Settings are set up at the ScrapyCloud project or spider level, and can be set in the console (shown in this article), or even by you settings.py during spider deploy. You can then access them like:

 

self.settings.getbool('SOME_BOOLEAN_SETTING')

 

Login to post a comment