Zyte Support Center
How can we help you today?
Declare pure environment variables or custom settings
Was this question ever answered? We're having a similar issue.
Hi Sam, this is Sergey Geron, Business Analyst at Zyte.
Thank you for reaching out through our Forum.
Could you please create a support ticket here - https://support.zyte.com/support/tickets/new
Hey, this is a very old issue. Was it resolved?
I am not sure how you are trying to access settings, but you should be able to access settings defined in Scrapy Cloud (or settings.py, YourSpider.custom_settings, etc.) by:
- In spiders: using self.settings.
- In Scrapy components (middlewares, pipelines, extensions, etc.): defining a from_crawler or from_settings class method, and passing crawler.settings or settings respectively to the __init__ method of your component in the implementation of that method. I recommend finding existing from_crawler or from_settings implementation in the Scrapy source code to get the idea.
- If none of that fits your use case, e.g. if your are trying to read Scrapy Cloud settings from a script, use the sh_scrapy.utils.get_project_settings function.
Last sh_scrapy modules worked for me!
Rafael Vaquero
I want to add some secret values to my crawler (database URI and api keys)
I don't want to add this values to the github repository, but I still want to use Scrapinghub.
I know that I can write custom settings and that this settings in the UI are overwriting my settings in settings.py, but.. in this case I could add the secret values to the UI in settings.
However if I do so, I still getting the variable is None, I have tried to load the settings after generate the start_urls, but it still empty.
How can I add legit environment variables or at least secret values in the config??
I assume I'm not the only one with this issue
2 people have this question