通过Google Cloud Deployment Manager创建的Google数据流 [英] Google dataflow created via google cloud deployment manager

问题描述

我正在尝试通过Deployment Manager创建带有模板Pub/Sub到BigQuery的数据流作业. 问题是运行gcloud deployment-manager types list命令后我看不到任何数据流服务.

I'm trying to create a dataflow job with template Pub/Sub to BigQuery via deployment manager. The problem is that I don't see any dataflow service after running gcloud deployment-manager types list command.

是否可以有一个Yaml文件来创建数据流作业?

Is it possible to have an yaml file to create dataflow job?

推荐答案

我将根据@lukaszberwid的回答为您提供一个如何为Cloud Deployment Manager设置数据流的工作示例.

I'm going to provide you with a working sample of how to setup Dataflow for Cloud Deployment Manager, based on the answer from @lukaszberwid.

首先，您需要一个options.yaml文件，以设置验证(此外，您可能需要在环境中设置服务帐户)

First you need an options.yaml file, to setup the auth (additionally you may need to setup your service account within your env):

options:
  inputMappings:
  - fieldName: Authorization
    location: HEADER
    value: '$.concat("Bearer ", $.googleOauth2AccessToken())'

然后，注册数据流运行:

Then, to register dataflow run:

gcloud beta deployment-manager type-providers create dataflow \
    --descriptor-url='https://dataflow.googleapis.com/$discovery/rest?version=v1b3' \
    --api-options-file=options.yaml

接下来，您可以找到示例resources.yaml，该示例可以从入门

Next you can find a sample resources.yaml that can launch the WordCount template from the get started guide:

resources:
  - name: "datafow-job"
    type: "[PROJECT-ID]/dataflow:dataflow.projects.locations.templates.launch"
    properties:
      projectId: [PROJECT-ID]
      location: us-central1
      gcsPath: "gs://dataflow-templates-us-central1/latest/Word_Count"
      jobName: testing-job
      environment:
        bypassTempDirValidation: false
        tempLocation: "gs://[BUCKET-NAME]/dataflow/tmp"
        ipConfiguration: "WORKER_IP_UNSPECIFIED"
        additionalExperiments: []
      parameters:
        inputFile: "gs://dataflow-samples/shakespeare/kinglear.txt"
        output: "gs://[BUCKET-NAME]/dataflow/output/my_output"

要运行它:

gcloud deployment-manager deployments create my-deployment \
    --config resources.yaml

这篇关于通过Google Cloud Deployment Manager创建的Google数据流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！