如何写入运行时定义的文件名? [英] How to write to a file name defined at runtime?

问题描述

我想写入gs文件，但是在编译时我不知道文件名.其名称基于运行时定义的行为.我该怎么办?

I want to write to a gs file but I don’t know the file name at compile time. Its name is based on behavior that is defined at runtime. How can I proceed?

推荐答案

如果您使用的是Beam Java，则可以为此使用FileIO.writeDynamic()(从当前正在发布的Beam 2.3开始-但是您可以可以通过版本2.3.0-SNAPSHOT)或较旧的DynamicDestinations API(在Beam 2.2中可用)使用它.

If you're using Beam Java, you can use FileIO.writeDynamic() for this (starting with Beam 2.3 which is currently in the process of being released - but you can already use it via the version 2.3.0-SNAPSHOT), or the older DynamicDestinations API (available in Beam 2.2).

根据交易类型，使用FileIO.writeDynamic()将银行交易PCollection写入GCS上不同路径的示例:

Example of using FileIO.writeDynamic() to write a PCollection of bank transactions to different paths on GCS depending on the transaction's type:

PCollection<BankTransaction> transactions = ...;
transactions.apply(
    FileIO.<BankTransaction, TransactionType>writeDynamic()
      .by(Transaction::getType)
      .via(BankTransaction::toString, TextIO.sink())
      .to("gs://bucket/myfolder/")
      .withNaming(type -> defaultNaming("transactions_", ".txt"));

有关DynamicDestinations使用的示例，请参见

For an example of DynamicDestinations use, see example code in the TextIO unit tests.

或者，如果要将每个记录写入其自己的文件，只需使用DoFn中的FileSystems API(特别是FileSystems.create()).

Alternatively, if you want to write each record to its own file, just use the FileSystems API (in particular, FileSystems.create()) from a DoFn.

这篇关于如何写入运行时定义的文件名?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！