在 BigQueryIO.write() 操作之后完全执行一个过程 [英] Execute a process exactly after BigQueryIO.write() operation

问题描述

我有一个带有 BigQuery 表作为接收器的管道.在将数据写入 BigQuery 之后，我需要完全执行一些步骤.这些步骤包括对该表执行查询、从中读取数据并写入不同的表.

I have a pipeline with a BigQuery table as sink. I need to perform some steps exactly after data has been written to BigQuery. Those steps include performing queries on that table, read data from it and write to a different table.

如何实现上述目标?我是否应该为后者创建一个不同的管道，然后在第一个管道之后调用它，这将是我认为的另一个问题.

How to achieve the above? Should I create a different pipeline for the latter but then calling it after the 1st pipeline will be another problem I assume.

如果以上都不起作用，是否可以从正在运行的管道中调用另一个数据流作业(模板).

If none of the above work, is it possible to call another dataflow job(template) from a running pipeline.

真的需要一些帮助.

谢谢.

推荐答案

BigQueryIO 目前未明确支持此功能.唯一的解决方法是使用单独的管道:启动第一个管道，等待它完成(例如使用 pipeline.run().waitUntilFinish())，启动第二个管道(确保使用一个单独的 Pipeline 对象 - 不支持多次重用同一个对象).

This is currently not explicitly supported by BigQueryIO. The only workaround is to use separate pipelines: start the first pipeline, wait for it to finish (eg. using pipeline.run().waitUntilFinish()), start the second pipeline (make sure to use a separate Pipeline object for it - reusing the same object multiple times is not supported).

这篇关于在 BigQueryIO.write() 操作之后完全执行一个过程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！