Flink 能否将结果写入多个文件(如 Hadoop 的 MultipleOutputFormat)? [英] Can Flink write results into multiple files (like Hadoop's MultipleOutputFormat)?

问题描述

我正在使用 Apache Flink 的数据集 API.我想实现一个将多个结果写入不同文件的作业.

I'm using Apache Flink's DataSet API. I want to implement a job that writes multiple results into different files.

我该怎么做?

推荐答案

您可以根据需要向 DataSet 程序添加任意数量的数据接收器.

You can add as many data sinks to a DataSet program as you need.

例如在这样的程序中:

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

DataSet<Tuple3<String, Long, Long>> data = env.readFromCsv(...);
// apply MapFunction and emit
data.map(new YourMapper()).writeToText("/foo/bar");
// apply FilterFunction and emit
data.filter(new YourFilter()).writeToCsv("/foo/bar2");

您从 CSV 文件中读取了 DataSet data.这个 data 被提供给两个后续的转换:

You read a DataSet data from a CSV file. This data is given to two subsequent transformations:

1. 到 MapFunction 并将其结果写入文本文件.
2. 到一个 FilterFunction 并且未过滤的元组被写入一个 CSV 文件.

1. To a MapFunction and its result is written to a text file.
2. To a FilterFunction and the non-filtered tuples are written to a CSV file.

您还可以拥有多个数据源和分支合并数据集(使用union、join、coGroup、cross，或广播集)，随你喜欢.

You can also have multiple data source and branch and merge data sets (using union, join, coGroup, cross, or broadcast sets) as you like.

这篇关于Flink 能否将结果写入多个文件(如 Hadoop 的 MultipleOutputFormat)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！