是否可以将Flume头文件写入HDFS接收器并放下主体？ [英] Is it possible to write Flume headers to HDFS sink and drop the body?

问题描述

text_with_headers序列化程序（HDFS汇序列化程序）允许保存Flume事件标题而不是丢弃它们。输出格式由标题，后面跟一个空格，然后是主体有效载荷组成。我们想放弃身体并仅保留标题。
对于HBase接收器，RegexHbaseEventSerializer允许我们转换事件。但是我无法为HDFS接收器找到这样的规定。

您可以设置序列化器属性添加到 header_and_text ，它将输出标题和正文。

例如：

  agent.sinks.my-hdfs-sink.type = hdfs 
 agent.sinks.my-hdfs-sink.hdfs.fileType = DataStream 
 .. 。
＃非常重要
 agent.sinks.my-hdfs-sink.serializer = header_and_text 
  

The text_with_headers serializer (HDFS sink serializer) allows to save the Flume event headers rather than discarding them. The output format consists of the headers, followed by a space, then the body payload. We would like to drop the body and retain the headers only. For the HBase sink, the "RegexHbaseEventSerializer" allows us to transform the events. But I am unable to find such a provision for the HDFS sink.

解决方案

You can set serializer property to header_and_text, which outputs both the headers and the body.

For example:

agent.sinks.my-hdfs-sink.type = hdfs
agent.sinks.my-hdfs-sink.hdfs.fileType = DataStream
... 
# very important
agent.sinks.my-hdfs-sink.serializer = header_and_text

这篇关于是否可以将Flume头文件写入HDFS接收器并放下主体？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！