使用Flume压缩文件摄取 [英] Compressed file ingestion using Flume

问题描述

使用Flume ng 1.3.0可以将任何类型的压缩文件（比如zip，bzip，lz4等）吸收到hdfs吗？我打算使用spoolDir。任何建议请。

解决方案

您可以摄取任何类型的文件。您需要选择一个合适的解串器。

下面的路线适用于压缩文件。您可以根据需要选择选项：

  agent.sources = src-1 
 agent.channels = c1 
 agent.sinks = k1 
 
 agent.sources.src-1.type = spooldir 
 agent.sources.src-1.channels = c1 
 agent.sources。 src-1.spoolDir = / tmp / myspooldir 
 agent.sources.src-1.deserializer = org.apache.flume.sink.solr.morphline.BlobDeserializer $ Builder 
 
 agent.channels .c1.type = file 
 
 agent.sinks.k1.type = hdfs 
 agent.sinks.k1.channel = c1 
 agent.sinks.k1.hdfs.path = / user / myevents / 
 agent.sinks.k1.hdfs.filePrefix = events- 
 agent.sinks.k1.hdfs.fileType = CompressedStream 
 agent.sinks.k1.hdfs.round = true 
 agent.sinks.k1.hdfs.roundValue = 10 
 agent.sinks.k1.hdfs.roundUnit =分钟
 agent.sinks.k1.hdfs.codeC = snappyCodec 
  

Can I ingest any type of compressed file ( say zip, bzip, lz4 etc.) to hdfs using Flume ng 1.3.0? I am planning to use spoolDir. Any suggesion please.

解决方案

You can ingest any type of file. You need to select an appropriate deserializer.

Below route works for compressed files. You can choose the options as you need:

agent.sources = src-1
agent.channels = c1
agent.sinks = k1

agent.sources.src-1.type = spooldir
agent.sources.src-1.channels = c1
agent.sources.src-1.spoolDir = /tmp/myspooldir
agent.sources.src-1.deserializer=org.apache.flume.sink.solr.morphline.BlobDeserializer$Builder

agent.channels.c1.type = file

agent.sinks.k1.type = hdfs
agent.sinks.k1.channel = c1
agent.sinks.k1.hdfs.path = /user/myevents/
agent.sinks.k1.hdfs.filePrefix = events-
agent.sinks.k1.hdfs.fileType = CompressedStream
agent.sinks.k1.hdfs.round = true
agent.sinks.k1.hdfs.roundValue = 10
agent.sinks.k1.hdfs.roundUnit = minute
agent.sinks.k1.hdfs.codeC = snappyCodec

这篇关于使用Flume压缩文件摄取的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！