如何写入在运行时定义的文件名? [英] How to write to a file name defined at runtime?

问题描述

我想写入一个 gs 文件，但在编译时我不知道文件名.它的名称基于在运行时定义的行为.我该如何继续?

I want to write to a gs file but I don’t know the file name at compile time. Its name is based on behavior that is defined at runtime. How can I proceed?

推荐答案

如果你使用 Beam Java，你可以使用 FileIO.writeDynamic() 来解决这个问题(从 Beam 2.3 开始，它目前是在发布过程中 - 但您已经可以通过版本 2.3.0-SNAPSHOT) 或较旧的 DynamicDestinations API(在 Beam 2.2 中可用)使用它.

If you're using Beam Java, you can use FileIO.writeDynamic() for this (starting with Beam 2.3 which is currently in the process of being released - but you can already use it via the version 2.3.0-SNAPSHOT), or the older DynamicDestinations API (available in Beam 2.2).

使用FileIO.writeDynamic()根据交易类型将银行交易的PCollection写入GCS上不同路径的示例:

Example of using FileIO.writeDynamic() to write a PCollection of bank transactions to different paths on GCS depending on the transaction's type:

PCollection<BankTransaction> transactions = ...;
transactions.apply(
    FileIO.<BankTransaction, TransactionType>writeDynamic()
      .by(Transaction::getType)
      .via(BankTransaction::toString, TextIO.sink())
      .to("gs://bucket/myfolder/")
      .withNaming(type -> defaultNaming("transactions_", ".txt"));

有关DynamicDestinations 使用的示例，请参阅TextIO 单元测试中的示例代码.

For an example of DynamicDestinations use, see example code in the TextIO unit tests.

或者，如果您想将每条记录写入自己的文件，只需使用 FileSystems API(特别是 FileSystems.create())来自 >DoFn.

Alternatively, if you want to write each record to its own file, just use the FileSystems API (in particular, FileSystems.create()) from a DoFn.

这篇关于如何写入在运行时定义的文件名?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！