Flink可以将结果写入多个文件(例如Hadoop的MultipleOutputFormat)吗? [英] Can Flink write results into multiple files (like Hadoop's MultipleOutputFormat)?

问题描述

我正在使用Apache Flink的DataSet API.我想实现一项将多个结果写入不同文件的作业.

I'm using Apache Flink's DataSet API. I want to implement a job that writes multiple results into different files.

我该怎么办?

推荐答案

您可以根据需要向DataSet程序添加尽可能多的数据接收器.

You can add as many data sinks to a DataSet program as you need.

例如在这样的程序中:

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();

DataSet<Tuple3<String, Long, Long>> data = env.readFromCsv(...);
// apply MapFunction and emit
data.map(new YourMapper()).writeToText("/foo/bar");
// apply FilterFunction and emit
data.filter(new YourFilter()).writeToCsv("/foo/bar2");

您从CSV文件中读取了DataSet data.此data提供了两个后续转换:

You read a DataSet data from a CSV file. This data is given to two subsequent transformations:

1. 发送到MapFunction，并将其结果写入文本文件.
2. 向FilterFunction并将未过滤的元组写入CSV文件.

1. To a MapFunction and its result is written to a text file.
2. To a FilterFunction and the non-filtered tuples are written to a CSV file.

您还可以根据需要选择多个数据源以及分支和合并数据集(使用union，join，coGroup，cross或广播集).

You can also have multiple data source and branch and merge data sets (using union, join, coGroup, cross, or broadcast sets) as you like.

这篇关于Flink可以将结果写入多个文件(例如Hadoop的MultipleOutputFormat)吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！