Flume HDFS接收器保持滚动的小文件 [英] Flume HDFS sink keeps rolling small files

问题描述

我试图使用flume将twitter数据转换为hdfs： https：/ /github.com/cloudera/cdh-twitter-example/

无论我在这里尝试什么，它都会在HDFS中创建大小不等的文件，大小从1.5kB到15kB，我希望看到大文件（64Mb）。
以下是代理配置：

  TwitterAgent.sources = Twitter 
 TwitterAgent.channels = MemChannel 
 TwitterAgent.sinks = HDFS 
 
 TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource 
 TwitterAgent.sources.Twitter.channels = MemChannel 
 TwitterAgent .sources.Twitter.consumerKey = xxxxxx 
 TwitterAgent.sources.Twitter.consumerSecret = xxxxxx 
 TwitterAgent.sources.Twitter.accessToken = xxxxx 
 TwitterAgent.sources.Twitter.accessTokenSecret = xxxxxxxx 
 TwitterAgent.sources.Twitter.keywords =测试
 
 TwitterAgent.sinks.HDFS.channel = MemChannel 
 TwitterAgent.sinks.HDFS.type = hdfs 
 TwitterAgent.sinks.HDFS。 hdfs.path = hdfs：//localhost.localdomain：8020 / user / flume / tweets /％Y /％m /％d /％H / 
 TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream 
 TwitterAgent.sinks.HDFS.hdfs.writeFormat =文本
 TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000 
 TwitterAgent.sinks.HDFS.hdfs.rollSize = 6 7108864 
 TwitterAgent.sinks.HDFS.hdfs.rollCount = 0 
 TwitterAgent.sinks.HDFS.hdfs.rollInterval = 0 
 TwitterAgent.sinks.HDFS.hdfs.idleTimeout = 0 
 
 TwitterAgent.channels.MemChannel.type =内存
 TwitterAgent.channels.MemChannel.capacity = 10000 
 TwitterAgent.channels.MemChannel.transactionCapacity = 1000 
  
 
 
 
编辑：我查看了日志文件，发现这一切都在发生： 
 
 
 
 
 9：11：27.526 AM WARN org.apache.flume.sink.hdfs.BucketWriter块
检测到欠复制。旋转文件。 9：11：37.036 AM错误
 org.apache.flume.sink.hdfs.BucketWriter 
 
 $ b 
击中最大连续复制旋转（30）;将不会
继续在此路径下滚动文件，因为复制不足 
 
 
 
解决方案
这似乎是HDFS复制因素的问题。由于我在使用1个虚拟datanode的虚拟机上工作，我必须将复制因子设置为1才能按预期工作。
 
I'm trying to stream twitter data into hdfs using flume and this: https://github.com/cloudera/cdh-twitter-example/

Whatever I try here, it keeps creating files in HDFS that range in size from 1.5kB to 15kB where I would like to see large files (64Mb).
Here is the agent configuration:
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = xxxxxx
TwitterAgent.sources.Twitter.consumerSecret = xxxxxx
TwitterAgent.sources.Twitter.accessToken = xxxxx
TwitterAgent.sources.Twitter.accessTokenSecret = xxxxxxxx
TwitterAgent.sources.Twitter.keywords = test

TwitterAgent.sinks.HDFS.channel = MemChannel
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost.localdomain:8020/user/flume/tweets/%Y/%m/%d/%H/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 10000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 67108864
TwitterAgent.sinks.HDFS.hdfs.rollCount = 0
TwitterAgent.sinks.HDFS.hdfs.rollInterval = 0
TwitterAgent.sinks.HDFS.hdfs.idleTimeout = 0

TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 1000
EDIT: I looked into the log files and found this happening all the time:

  
9:11:27.526 AM    WARN    org.apache.flume.sink.hdfs.BucketWriter     Block
  Under-replication detected. Rotating file. 9:11:37.036 AM     ERROR
    org.apache.flume.sink.hdfs.BucketWriter     
  
  
Hit max consecutive under-replication rotations (30); will not
  continue rolling files under this path due to under-replication

解决方案
It seemed to be a problem with the HDFS replication factor. As I am working on a virtual machine with 1 virtual datanode I had to set the replication factor to 1 in order for it to work as expected.

                        
这篇关于Flume HDFS接收器保持滚动的小文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！