恰好在BigQueryIO.write()操作之后执行一个过程 [英] Execute a process exactly after BigQueryIO.write() operation

问题描述

我有一个使用BigQuery表作为接收器的管道.在将数据写入BigQuery之后，我需要准确地执行一些步骤.这些步骤包括在该表上执行查询，从该表读取数据并写入另一个表.

I have a pipeline with a BigQuery table as sink. I need to perform some steps exactly after data has been written to BigQuery. Those steps include performing queries on that table, read data from it and write to a different table.

如何实现以上目标?我应该为后者创建一个不同的管道，但在第一个管道之后调用它是另一个问题.

How to achieve the above? Should I create a different pipeline for the latter but then calling it after the 1st pipeline will be another problem I assume.

如果以上方法均无效，则可以从正在运行的管道中调用另一个数据流作业(模板).

If none of the above work, is it possible to call another dataflow job(template) from a running pipeline.

真的需要一些帮助.

谢谢.

推荐答案

BigQueryIO当前未明确支持.唯一的解决方法是使用单独的管道:启动第一个管道，等待其完成(例如，使用 pipeline.run().waitUntilFinish())，启动第二个管道(确保使用一个单独的Pipeline对象-不支持重复使用同一对象多次.

This is currently not explicitly supported by BigQueryIO. The only workaround is to use separate pipelines: start the first pipeline, wait for it to finish (eg. using pipeline.run().waitUntilFinish()), start the second pipeline (make sure to use a separate Pipeline object for it - reusing the same object multiple times is not supported).

这篇关于恰好在BigQueryIO.write()操作之后执行一个过程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！