Start a new topic

Declare pure environment variables or custom settings

I want to add some secret values to my crawler (database URI and api keys)


I don't want to add this values to the github repository, but I still want to use Scrapinghub.


I know that I can write custom settings and that this settings in the UI are overwriting my settings in settings.py, but.. in this case I could add the secret values to the UI in settings.


However if I do so, I still getting the variable is None, I have tried to load the settings after generate the start_urls, but it still empty.


How can I add legit environment variables or at least secret values in the config??


I assume I'm not the only one with this issue


2 people have this question

Was this question ever answered? We're having a similar issue.

Hi Sam, this is Sergey Geron, Business Analyst at Zyte.
Thank you for reaching out through our Forum.


Could you please create a support ticket here - https://support.zyte.com/support/tickets/new

Hey, this is a very old issue. Was it resolved?

I am not sure how you are trying to access settings, but you should be able to access settings defined in Scrapy Cloud (or settings.py, YourSpider.custom_settings, etc.) by:


- In spiders: using self.settings.

- In Scrapy components (middlewares, pipelines, extensions, etc.): defining a from_crawler or from_settings class method, and passing crawler.settings or settings respectively to the __init__ method of your component in the implementation of that method. I recommend finding existing from_crawler or from_settings implementation in the Scrapy source code to get the idea.

- If none of that fits your use case, e.g. if your are trying to read Scrapy Cloud settings from a script, use the sh_scrapy.utils.get_project_settings function.


1 person likes this

Last sh_scrapy modules worked for me!

Login to post a comment