如何获取使用Dataproc工作流模板提交的jobId [英] How to get jobId that was submitted using Dataproc Workflow Template

问题描述

我已经在Airflow操作员（用Python编写的DataprocWorkflowTemplateInstantiateInlineOperator ）。提交作业后，会将一些名称指定为jobId（例如： job0-abc2def65gh12 ）。

I have submitted a Hive job using Dataproc Workflow Template with the help of Airflow operator (DataprocWorkflowTemplateInstantiateInlineOperator) written in Python. Once the job is submitted some name will be assigned as jobId (example: job0-abc2def65gh12).

无法获取jobId，我试图通过不起作用的REST API传递jobId作为参数。

Since I was not able to get jobId I tried to pass jobId as a parameter from REST API which isn't working.

我可以获取jobId还是，如果无法获取，可以我将jobId作为参数传递吗？

Can I fetch jobId or, if it's not possible, can I pass jobId as a parameter?

推荐答案

JobId将作为元数据 Operation 对象中的c $ c>字段。有关如何使用元数据，请参见此[1]文章。

The JobId will be available as part of metadata field in Operation object that is returned from Instantiate operation. See this [1] article for how to work with metadata.

Airflow操作员仅对Operation轮询[2]，但不返回最终的Operation对象。您可以尝试在 execute 中添加返回值。

The Airflow operator only polls [2] on the Operation but does not return the final Operation object. You could try to add a return to execute.

另一种选择是使用dataproc rest API [3 ]在工作流程完成之后。分配给工作流本身的所有标签都将传播到群集和作业，因此您可以进行列表作业调用。例如，filter参数可能类似于： filter = labels.my-label = 12345

Another option would to be to use dataproc rest API [3] after workflow finishes. Any labels assigned to the workflow itself will be propagated to clusters and jobs so you can do a list jobs call. For example the filter parameter could look like: filter = labels.my-label=12345

[1] < a href = https://cloud.google.com/dataproc/docs/concepts/workflows/debugging#using_workflowmetadata rel = nofollow noreferrer> https://cloud.google.com/dataproc/docs/concepts/workflows / debugging＃using_workflowmetadata

[2] https://github.com/apache/airflow/blob/master/airflow/contrib/operators/dataproc_operator.py#L1376

[3] https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.jobs/list

这篇关于如何获取使用Dataproc工作流模板提交的jobId的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！