在GCP Cloudbuild中安装python要求时出现问题 [英] Problem with Installing python requirements in GCP cloudbuild

问题描述

我正在尝试在Cloudbuild中使用DirectRunner运行apache Beam管道，并且这样做需要安装python脚本的要求，但是我遇到了一些错误.
这是我cloudbuild.yaml的一部分

I am trying to run a apache beam pipeline with DirectRunner in cloudbuild and by doing that I need to install the requirements for the python script, but I am facing some errors.
This is part of my cloudbuild.yaml

    steps:
- name: gcr.io/cloud-builders/gcloud
  entrypoint: 'bash'
  args: [ '-c', "gcloud secrets versions access latest --secret=env --format='get(payload.data)' | tr '_-' '/+' | base64 -d > .env" ]
  id: GetSecretEnv
# - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
#   entrypoint: 'bash'
#   args: ['-c', 'gcloud config set app/cloud_build_timeout 1600 && gcloud app deploy --quiet tweepy-to-pubsub/app.yaml']

- name: gcr.io/cloud-builders/gcloud
  id: Access id_github
  entrypoint: 'bash'
  args: [ '-c', 'gcloud secrets versions access latest --secret=id_github> /root/.ssh/id_github' ]
  volumes:
  - name: 'ssh'
    path: /root/.ssh
# Set up git with key and domain
- name: 'gcr.io/cloud-builders/git'
  id: Set up git with key and domain
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    chmod 600 /root/.ssh/id_github
    cat <<EOF >/root/.ssh/config
    Hostname github.com
    IdentityFile /root/.ssh/id_github
    EOF
    ssh-keyscan -t rsa github.com > /root/.ssh/known_hosts
  volumes:
  - name: 'ssh'
    path: /root/.ssh
- name: 'gcr.io/cloud-builders/git'
# Connect to the repository
  id: Connect and clone repository
  dir: workspace
  args:
  - clone
  - --recurse-submodules
  - [e-mail]:x/repo.git
  volumes:
  - name: 'ssh'
    path: /root/.ssh

- name: 'gcr.io/$PROJECT_ID/dataflow-python3'
  entrypoint: '/bin/bash'
  args: [ '-c',
          'source /venv/bin/activate' ]
- name: 'gcr.io/$PROJECT_ID/dataflow-python3'  
  entrypoint: '/bin/bash' 
  dir: workspace
  args: ['pip', 'install','-r', '/dir1/dir2/requirements.txt']
- name: 'gcr.io/$PROJECT_ID/dataflow-python3'
  entrypoint: 'python'
  dir: workspace
  args: [ 'dir1/dir2/script.py', 
         '--runner=DirectRunner' ]
timeout: "1600s"

没有安装要求的步骤，它可以工作，但是我需要库，因为我缺少缺少库的python错误，第二步(实际上是云构建的原始形式的第5步)，云构建因此失败错误

Without the step where I install the requirements this works but I need the libs, because I have python error for missing libs, and on the second step (5th actually in the original form of cloud build) the cloud build fails with this error

Step #5: Already have image (with digest): gcr.io/x/dataflow-python3
Step #5: import-im6.q16: unable to open X server `' @ error/import.c/ImportImageCommand/360.
Step #5: import-im6.q16: unable to open X server `' @ error/import.c/ImportImageCommand/360.
Step #5: /usr/local/bin/pip: line 5: from: command not found
Step #5: /usr/local/bin/pip: pip: line 7: syntax error near unexpected token `('
Step #5: /usr/local/bin/pip: pip: line 7: `    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])'

我该如何解决?我还在互联网上尝试了一些示例，但它不起作用

How do I fix this? I also tried some examples on the internet and it doesn't work

首先，我在应用程序引擎上进行部署，然后在云构建虚拟机中下载存储库，安装需求并尝试使用python脚本运行它

First I deploy on app engine and then I download the repo in cloud build vm, install requirements and try to run it the python script

推荐答案

我认为问题出在您的路径定义中

I think that the issue comes from your path definition

'source /venv/bin/activate'

和

'pip', 'install','-r', '/dir1/dir2/requirements.txt'

您使用完整路径定义，并且在Cloud Build上不起作用.当前工作目录为/workspace/.如果您使用相对路径，只需在路径前面添加一个点.，它应该会更好.

You use the full path definition and it doesn't work on Cloud Build. The current working directory is /workspace/. If you use relative path, add simply a dot . before the path, it should works better.

还是不......确实，您一步完成了venv激活，然后在接下来的步骤中安装了pip.从一个步骤到另一个步骤，将运行时环境卸载并与另一个容器一起重新加载.因此，设置环境变量的 source 命令在 pip 步骤中消失.

Or not... Indeed, you have the venv activation in a step, and the pip install in the following step. From one step to another, the runtime environment is offloaded and reloaded with the other container. Thus, your source command that set up environment variable, disappear in the pip step.

此外，您的云构建环境是为构建而构建的，然后销毁.在这种情况下，您无需使用venv，就可以简化最后3个步骤

In addition, your cloud build environment is built for the build and destroy then. You don't need to use venv in this case and you can simplify the 3 last steps like this

- name: 'gcr.io/$PROJECT_ID/dataflow-python3'
  entrypoint: '/bin/bash'
  args:
   - '-c'
   - |
       pip install -r ./dir1/dir2/requirements.txt
       python ./dir1/dir2/script.py --runner=DirectRunner

这篇关于在GCP Cloudbuild中安装python要求时出现问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！