在部署到gcloud的应用程序中链接packages.json中的私有存储库 [英] Link private repository in packages.json in app deployed to gcloud

问题描述

我有一个我要为其设置持续部署的node.js应用程序(my-app).该存储库托管在Bitbucket上，我正在使用Bitbucket的Pipelines创建CD.管道脚本将应用程序部署到Google Cloud Platform和App Engine.

I have a node.js app (my-app) that I'm setting Continuous Deployment for. The repository is hosted on Bitbucket and I'm creating CD with Bitbucket's Pipelines. The pipelines script deploys the app to Google Cloud Platform, to App Engine.

当我尝试向Bitbucket上另一个私有存储库中的my-app添加程序包(my-package)时遇到问题.这是我的packages.json文件的一部分:

I am encountering a problem when I try to add a package (my-package) to my-app that is in another private repository on Bitbucket. Here is an part of my packages.json file:

"dependencies": {
    "my-package": "git+ssh://[e-mail]:team-name/my-package.git",
}

通过这种链接，我能够:

With such linking I was able to:

1. 将我的PC的ssh密钥添加到存储库中，以确保npm install在本地运行
2. 将管道ssh密钥添加到存储库中，以确保npm install在管道环境中正常工作

1. Add my PC's ssh key to repository to make sure npm install works locally
2. Add pipelines ssh key to repository to make sure that npm install works inside pipelines environment

我正在努力的是管道脚本中的gcloud app deploy命令.据我所知，此命令在部署node.js应用程序时在Google的环境中运行npm install.这样的环境无法使用my-package访问我的私有存储库，并随后引发此错误:

What I'm struggling with is the gcloud app deploy command inside the pipelines script. To my best knowledge, this command, when deploying a node.js app, runs npm install inside Google's environment. Such environment doesn't have access to my private repository with my-package and subsequently throws this error:

ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build ##### status: FAILURE.
(...)
INFO     gen_package_lock npm install --package-lock-only
INFO     `gen_package_lock` stdout:
INFO     `gen_package_lock` had stderr output:
npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://[e-mail]:team-name/my-package.git
npm ERR! 
npm ERR! Host key verification failed.
npm ERR! fatal: Could not read from remote repository.
npm ERR! 
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR! 
npm ERR! exited with error code: 128

对我来说，显而易见的解决方案是在Google环境中生成ssh密钥并授予对私有存储库访问权限的可能性.

The obvious solution for me would be a possibility to generate an ssh key inside the Google's environment and granting access to the private repository.

有可能吗?如果没有，在这里可以采用什么其他解决方案?

Is it possible? If not - what other solutions could be applied here?

推荐答案

我能够使其正常运行，并将很高兴地与您分享我已采取的步骤.我已经在我的 GitHub 中发布了一个工作示例，因此您可以对其进行克隆并对其进行尝试您自己，请注意，一周后我将删除此内容.我将从头开始进行整个过程，即使您已经设置了SSH密钥，我也建议您这样做.

I was able to make it work and will happily share with you the steps that I’ve taken. I’ve posted a working example in my GitHub so you can clone it and try it out for yourself, do note that I will be deleting this after a week. I will be going through the whole process from scratch and I encourage you to do so, even if you already have your SSH keys set up.

1. 首先，将 GitHub存储库和cd克隆到其中.
2. 创建一个云KMS KeyRing和CryptoKey.假设您已经安装了Cloud SDK，请运行gcloud kms keyrings create my-keyring --location=global创建密钥环.接下来，通过运行gcloud kms keys create bit-key --location=global --keyring=my-keyring --purpose=encryption创建CryptoKey.
3. 接下来，通过运行ssh-keygen -t rsa -b 4096 -C [e-mail]创建SSH密钥.每次按一次，它都会提示您输入一个释义和一个文件来保存密钥.
4. 将SSH密钥配置到Bitbucket中.转到"Bitbucket设置"下的bitbucket.org，然后在选项之一中单击"SSH密钥".添加一个密钥(将其标记为您决定的名称)，然后将通过运行cat ~/.ssh/id_rsa.pub获得的输出粘贴到密钥部分中.
5. 记下id_rsa文件的位置，并获取其相对于项目文件夹的路径.例如，由于我是在Cloud Shell中进行复制的，因此我的.ssh/id_rsa文件相对于我的项目文件夹的路径就是../.ssh/id_rsa.然后通过运行gcloud kms encrypt --plaintext-file=<RELATIVE_PATH_TO_.SSH/id_rsa> --ciphertext-file=./id_rsa.enc --location=global --keyring=my-keyring --key=bit-key加密SSH密钥.在我的示例中，只是../.ssh/id_rsa
6. 首先进入console.cloud.google.com/iam-admin并找到Cloud Build服务帐户，以授予您的Cloud Build服务帐户解密权限:其名称类似于@ cloudbuild.gserviceaccount.com并将其授予应用程序引擎管理员角色(这样我们以后可以从Cloud Build运行gcloud应用程序部署)，然后复制该@ cloudbuild.gserviceaccount.com服务帐户的电子邮件地址，然后转到console.cloud.google.com/security/kms，然后选中我的密钥"复选框.在右侧面板上，您将看到权限，并在其中添加新成员并粘贴刚复制的服务帐户.向其授予Cloud KMS> Cloud KMS CryptoKey Decrypter角色.
7. 通过运行ssh-keyscan -t rsa bitbucket.org > known_hosts
创建或更新known_hosts文件
8. 运行gcloud builds submit --config=cloudbuild.yaml

1. First and foremost, clone the GitHub repo and cd into it.
2. Create a cloud KMS KeyRing and CryptoKey. Assuming you have the Cloud SDK installed, run gcloud kms keyrings create my-keyring --location=global to create a keyring. Next, create the CryptoKey by running gcloud kms keys create bit-key --location=global --keyring=my-keyring --purpose=encryption.
3. Next, create an SSH key by running ssh-keygen -t rsa -b 4096 -C [e-mail]. It will prompt you for a paraphrase and a file to save the key, just press each time.
4. Configure your SSH key into Bitbucket. Go into bitbucket.org under "Bitbucket Settings" and then click "SSH Keys" in one of the options. Add a key (label it whichever name you decide) and paste the output you get from running cat ~/.ssh/id_rsa.pub into the key section.
5. Take note of where your id_rsa file is located and get its path relative to your project folder. For example, because I did this reproduction in the Cloud Shell, my .ssh/id_rsa file’s path relative to my project folder is simply ../.ssh/id_rsa. Then encrypt the SSH key by running gcloud kms encrypt --plaintext-file=<RELATIVE_PATH_TO_.SSH/id_rsa> --ciphertext-file=./id_rsa.enc --location=global --keyring=my-keyring --key=bit-key. In my example, would simply be ../.ssh/id_rsa
6. Grant your Cloud Build service account decrypt permission by first going into console.cloud.google.com/iam-admin and find the Cloud Build service account: its name is similar to @cloudbuild.gserviceaccount.com and grant it the App Engine Admin role (this is so that we are able to run gcloud app deploy from Cloud Build later on) then copy the email address of that @cloudbuild.gserviceaccount.com service account and go to console.cloud.google.com/security/kms and select the checkbox of ‘my-keyring’. On the panel to the right, you will see the permissions and there add a new member and paste the service account you just copied. Grant it the Cloud KMS > Cloud KMS CryptoKey Decrypter role.
7. Create or update the known_hosts file by running ssh-keyscan -t rsa bitbucket.org > known_hosts
8. Run gcloud builds submit --config=cloudbuild.yaml

如您所见，我在名为circular-structure-stringify的bitbucket上托管了一个私有存储库.从cloudbuild.yaml中，您将看到SSH密钥首先被解密为位于/root/.ssh/id_rsa中的纯文本，然后在下一步中用于通过Bitbucket设置密钥.

As you can see, I have a private repository hosted on bitbucket called circular-structure-stringify. From the cloudbuild.yaml, you will see that the SSH key is first decrypted into a plaintext located in /root/.ssh/id_rsa, which is then used in the next step to set up the key with Bitbucket.

接下来，我们将Bitbucket中的私有存储库克隆到我的容器中，然后是npm install和gcloud app deploy.您将注意到，依赖项现在与应用程序位于同一文件夹中，即package.json文件所在的文件夹.

Next, we clone the private repository from Bitbucket into my container, followed by an npm install and gcloud app deploy. As you will notice, the dependency now lives in the same folder as our application, which is the same folder where the package.json file is located.

最后，在您的package.json中，将依赖项添加到dependencies属性中，如下所示:dependencies: {circular-structure-stringify: ./circular-structure-stringify}.随后，您可以像在任何npm软件包中一样导入模块:const CircularStructureStringify = require(‘circular-structure-stringify’)就像在/routes/index.js文件中看到的一样.

Finally, in your package.json, add the dependency in the dependencies property like such: dependencies: {"circular-structure-stringify": "./circular-structure-stringify"}. Subsequently, you can import the module like you would in any npm packages: const CircularStructureStringify = require(‘circular-structure-stringify’) like you can see in the /routes/index.js file.

要回答您的问题，是的，有可能.您将必须使用Cloud KMS 1 与私有Bitbucket存储库进行交互.有一个文档说明了访问私有Github存储库[2]所需的步骤，但是必须对其进行些微调整以使其可以与Bitbucket一起使用.

To answer your question, yes, it is possible. You will have to use Cloud KMS 1 to interact with a private Bitbucket repository. There exists a documentation explaining the steps needed to access a private Github repository [2], but it must be adjusted slightly to make it work with Bitbucket.

此外，在生成SSH密钥时，请确保提供指定的-C"[e-mail]".根据过去的经验，如果密钥在创建时未设置此密钥(YMMV)，我会遇到Bitbucket的问题.您可以参考本文档[3]以获得逐步说明.

Furthermore, when generating an SSH key, make sure to provide the -C "[e-mail]" is specified. From past experiences, I’ve had issues with Bitbucket specifically if the key didn’t have this set upon creation (YMMV).. You can refer to this document [3] for step by step instructions.

另一种解决方案是将您的应用托管在私有存储库上，然后使用Google Cloud Source Repositories [4]镜像/克隆该存储库，运行npm install and deploy.如这篇StackOverflow文章[5]中所述，您将必须在根文件夹(app.yaml文件所在的文件夹)上创建一个cloudbuild.yaml文件:

Another solution would be to have your app hosted on the private repository and then mirror/clone that repository using Google Cloud Source Repositories [4], run npm install and deploy. As explained in this StackOverflow post, [5], you will have to create a cloudbuild.yaml file on the root folder (same folder the app.yaml file is located) :

steps:
# NPM install
- name: 'gcr.io/cloud-builders/npm'
  args: ['install']
#Test
- name: 'gcr.io/cloud-builders/npm'
  args: ['test']
#Deploy
- name: "gcr.io/cloud-builders/gcloud"
  args: ["app", "deploy"]

然后，您将不得不将私有Bitbucket存储库镜像到Cloud Source Repository [4]，创建一个Cloud Build Trigger，以在将新代码推送到存储库时自动进行部署，最后将包含您的应用程序的文件夹推送到存储库.

You will then have to mirror the private Bitbucket repository to Cloud Source Repository [4], create a Cloud Build Trigger to automate deployment when new code has been pushed to the repository, and then finally pushing the folder containing your application to the repository.

• 1 https://cloud.google. com/cloud-build/docs/access-private-github-repos
• [3] https://help.github.com/en/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
• [4] https://cloud.google. com/source-repositories/docs/mirroring-a-bitbucket-repository
• [5] https://cloud.google. com/cloud-build/docs/running-builds/automate-builds

• 1 https://cloud.google.com/kms/
• [2] https://cloud.google.com/cloud-build/docs/access-private-github-repos
• [3]https://help.github.com/en/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
• [4] https://cloud.google.com/source-repositories/docs/mirroring-a-bitbucket-repository
• [5]How to use private, self hosted NPM package with Google App Engine node, standard environment
• [6] https://cloud.google.com/cloud-build/docs/running-builds/automate-builds

这篇关于在部署到gcloud的应用程序中链接packages.json中的私有存储库的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！