从云功能启动云数据流 [英] Launching Cloud Dataflow from Cloud Functions

问题描述

如何从 Google Cloud Function 启动Cloud Dataflow作业?我想将Google Cloud Functions作为一种启用跨服务组合的机制.

How do I launch a Cloud Dataflow job from a Google Cloud Function? I'd like to use Google Cloud Functions as a mechanism to enable cross-service composition.

推荐答案

我提供了 WordCount 示例如下.请注意，您需要在Cloud Function部署中包括Java二进制文件的副本，因为它不在默认环境中.同样，您还需要将自己的部署jar与Cloud Function打包在一起.

I've included a very basic example of the WordCount sample below. Please note that you'll need to include a copy of the java binary in your Cloud Function deployment, since it is not in the default environment. Likewise, you'll need to package your deploy jar with your Cloud Function as well.

module.exports = {
  wordcount: function (context, data) {
    const spawn = require('child_process').spawn;
    const child = spawn(
            'jre1.8.0_73/bin/java',
            ['-cp',
             'MY_JAR.jar',
             'com.google.cloud.dataflow.examples.WordCount',
             '--jobName=fromACloudFunction',
             '--project=MY_PROJECT',
             '--runner=BlockingDataflowPipelineRunner',
             '--stagingLocation=gs://STAGING_LOCATION',
             '--inputFile=gs://dataflow-samples/shakespeare/*',
             '--output=gs://OUTPUT_LOCATION'
            ],
            { cwd: __dirname });

    child.stdout.on('data', function(data) {
      console.log('stdout: ' + data);
    });
    child.stderr.on('data', function(data) {
      console.log('error: ' + data);
    });
    child.on('close', function(code) {
      console.log('closing code: ' + code);
    });
    context.success();
  }
}

您可以通过使用非阻塞运行器进一步增强此示例该函数返回作业ID，以便您可以分别轮询以完成作业.该模式对于其他SDK也应有效，只要它们的依赖项可以打包到Cloud Function中即可.

You could further enhance this example by using the non-blocking runner and having the function return the Job ID, so that you can poll for job completion separately. This pattern should be valid for other SDKs as well, so long as their dependencies can be packaged into the Cloud Function.

这篇关于从云功能启动云数据流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！