It goes against my company's security policy to put a cloud service ID string in version control, so this means that we can't commit the necessary `scrapinghub.yml` into version control without using something like git-crypt to encrypt the file first.
We were hoping that you could use environment variables within `scrapinghub.yml` but it appears this isn't supported.
Is there known pattern or best practice approach for reading project IDs from environment variables or otherwise, so as to not commit them to source control?
0 Votes
1 Comments
J
Julio Villalta IIIposted
over 3 years ago
Hi Ned,
I believe that your asking about storing projectIds for the purpose of deploying your spiders to ScrapyCloud correct?
If so, I have found that it is not a strict requirement to store the project Id in the scrapinghub.yml to have a spider deploy.
In my case we manage multiple projects with the same spider. To deploy, I specify the projectId after my depploy command (using shub). So for example, for project 123456, it would be:
shub deploy 123456
So long as your user has access to a project and you login to shub with the API key for that user, then you can deploy to any of those projects. This then allows you to store those ids however you need. For example, in the case of using Jenkins to deploy, you could save as a build parameter or stored secret (or password).
It goes against my company's security policy to put a cloud service ID string in version control, so this means that we can't commit the necessary `scrapinghub.yml` into version control without using something like git-crypt to encrypt the file first.
We were hoping that you could use environment variables within `scrapinghub.yml` but it appears this isn't supported.
Is there known pattern or best practice approach for reading project IDs from environment variables or otherwise, so as to not commit them to source control?
0 Votes
1 Comments
Julio Villalta III posted over 3 years ago
Hi Ned,
I believe that your asking about storing projectIds for the purpose of deploying your spiders to ScrapyCloud correct?
If so, I have found that it is not a strict requirement to store the project Id in the scrapinghub.yml to have a spider deploy.
In my case we manage multiple projects with the same spider. To deploy, I specify the projectId after my depploy command (using shub). So for example, for project 123456, it would be:
So long as your user has access to a project and you login to shub with the API key for that user, then you can deploy to any of those projects. This then allows you to store those ids however you need. For example, in the case of using Jenkins to deploy, you could save as a build parameter or stored secret (or password).
0 Votes
Login to post a comment