My scrapy project makes use of certain environment variables. I want to migrate my project to Scrapy cloud and need to set environment variable on cloud environment. Can somebody suggest how to achieve the same.
4 Votes
1 Comments
J
Julio Villalta IIIposted
over 3 years ago
Hi,
As suggested by this post, by strict definition, the environment variables (i.e os.environ["FOO"]) is usually not the values you'll want to use for custom external configuration.
However you should use spider settings or arguments.
Arguments are those config values per execution of a spider, and are available in init() function of your spider, like:
class Spider(CrawlSpider):
name = 'foo'
def __init__(self, *a, **kw):
custom_setting=kw.get('some_setting')
Spider Settings are set up at the ScrapyCloud project or spider level, and can be set in the console (shown in this article), or even by you settings.py during spider deploy. You can then access them like:
My scrapy project makes use of certain environment variables. I want to migrate my project to Scrapy cloud and need to set environment variable on cloud environment. Can somebody suggest how to achieve the same.
4 Votes
1 Comments
Julio Villalta III posted over 3 years ago
Hi,
As suggested by this post, by strict definition, the environment variables (i.e os.environ["FOO"]) is usually not the values you'll want to use for custom external configuration.
However you should use spider settings or arguments.
Arguments are those config values per execution of a spider, and are available in init() function of your spider, like:
class Spider(CrawlSpider): name = 'foo' def __init__(self, *a, **kw): custom_setting=kw.get('some_setting')Spider Settings are set up at the ScrapyCloud project or spider level, and can be set in the console (shown in this article), or even by you settings.py during spider deploy. You can then access them like:
self.settings.getbool('SOME_BOOLEAN_SETTING')0 Votes
Login to post a comment