GCP 数据流计算图和作业执行 [英] GCP Dataflow Computation Graph and Job Execution

问题描述

大家好，当我在 Google Cloud Dataflow 中创建自定义模板时，我努力理解发生了什么，但未能理解.感谢 GCP 文档.以下是我正在实现的目标.

Hi Everyone I tried hard to understand what is happening when I create a custom template in Google cloud Dataflow but failed to understand. Thanks to GCP documentations. Below is what I am achieving.

1. 从 Google Cloud Bucket 读取数据
2. 预处理
3. 加载深度学习模型(每个 1 GB)并获得预测
4. 将结果转储到 BigQuery 中.

我成功创建了模板并且能够执行该作业.但我有以下问题.

I successfully created the template and I am able to execute the job. But I have following questions.

1. 当我执行作业时，每次在执行过程中下载模型(5 个模型，每个模型 1GB)OR 模型被加载并放置在模板(执行图)中，并在执行过程中使用加载的
2. 如果模型加载只在作业执行期间发生，那么它不会影响执行时间吗?由于每次触发作业时都必须加载 GB 的模型文件?
3. 多个用户可以同时触发同一个模板吗?由于我想生产它，我不确定这将如何同时处理多个请求?

1. When I execute the job, Everytime the models (5 models and each of 1GB) gets downloaded during execution OR the models are loaded and placed in the template (Execution Graph) and during execution it uses the loaded ones
2. If loading of the models happen only during the job execution, then does it not impact the execution time? Since it has to load GBs of Model files everytime the job is triggered?
3. Can multiple users trigger the same template at same time? Since I want to productionize it, I am not sure how this will handle multiple requests at same time?

有人可以分享一些关于它的信息吗?

Can anyone please share some information on it?

我参考但未能得到答案的来源:https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#pipeline-lifecycle-from-pipeline-code-to-dataflow-job http://alumni.media.mit.edu/~wad/magiceight/isa/node3.html https://cloud.google.com/dataflow/docs/guides/setting-pipeline-options#configuring-pipelineoptions-for-local-execution https://beam.apache.org/documentation/basics/ https://beam.apache.org/documentation/runtime/model/ https://mehmandarov.com/apache-beam-pipeline-graph/

Sources I referred and failed to get the answer: https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#pipeline-lifecycle-from-pipeline-code-to-dataflow-job http://alumni.media.mit.edu/~wad/magiceight/isa/node3.html https://cloud.google.com/dataflow/docs/guides/setting-pipeline-options#configuring-pipelineoptions-for-local-execution https://beam.apache.org/documentation/basics/ https://beam.apache.org/documentation/runtime/model/ https://mehmandarov.com/apache-beam-pipeline-graph/

推荐答案

这取决于加载模型的位置.如果它们被加载到 DoFns 中(最有可能)，那么它就会发生在工作器中(在作业执行期间).

This depends on where the models are being loaded from. If they're loaded in the DoFns (most likely), then it will happen in the workers (during job execution).

至于您的另一个问题，多个用户同时触发模板作业应该没有问题.

As for your other question, there should be no issues with multiple users triggering a template job simultaneously.

这篇关于GCP 数据流计算图和作业执行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！