NiFi moveHDFS 处理器似乎什么都不做 [英] NiFi moveHDFS processor appears to do nothing

问题描述

只是尝试自动将文件从一个 HDFS 目录移动到另一个.使用 Apache NiFi 中的 moveHDFS 处理器执行此操作，但在启动处理器时似乎没有任何反应.

Simply trying to automate a move of files from one HDFS dir to another. Doing this with a moveHDFS processor in Apache NiFi, but when starting the processor nothing seems to happen.

处理器指标在很长一段时间后保持为零并且查看公告板显示没有错误(日志级别设置为 INFO)，公告板中唯一的日志输出是:

The processor metrics remain at zero after long amount of time and looking at the bulletin board shows no errors (logging level set to INFO), the only logging output in the bulletin board is:

14:50:04 HSTINFO1e637d0d-0163-1000-7bde-a7993ae403e8
MoveHDFS[id=1e637d0d-0163-1000-7bde-a7993ae403e8] Initialized a new HDFS File System with working dir: file:/home/mapr/nifi-1.6.0 default block size: 33554432 default replication: 1 config: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml, /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/hdfs-site.xml

(顺便说一句，为什么会说初始化一个新的 HDFS 文件系统"?难道不应该使用提供的 hadoop 配置文件来访问 hadoop 集群上的现有 HDFS?).将 NiFi 作为单个实例安装在 hadoop 集群节点上，并根据需要将 core- 和 hdfs-site.xml 文件添加到处理器属性中.以前从未使用过 NiFi 并怀疑我在这里缺少正确配置流程的明显信息.任何关于这里发生的事情以及如何解决它的建议将不胜感激.谢谢.

(By the way, why does it say "Initialized a new HDFS File System"? Shouldn't it be using the provided hadoop config files to access the existing HDFS on the hadoop cluster?). Have NiFi installed as a single instance on a hadoop cluster node and added the core- and hdfs-site.xml files to the processor properties as needed. Have never used NiFi before and suspect that I am missing something obvious about properly configuring the flow here. Any advice about what is going on here and how to fix it would be appreciated. Thanks.

推荐答案

如果你能分享更多你的流程，那会很有帮助.MoveHDFS 处理器需要传入的流文件才能处理它们，因此无论处理器生成或接收流文件，都需要将它们传递给该处理器.队列是否显示任何排队的流文件?处理器显示在过去五分钟内没有读入任何流文件.您可以使用 ListHDFS 或 GenerateFlowFile 来创建该处理器期望的流文件.

If you can share more of your flow, that would be helpful. The MoveHDFS processor requires incoming flowfiles in order to process them, so whatever processor is generating or receiving flowfiles needs to pass them on to this processor. Does the queue show any queued flowfiles? The processor shows that no flowfiles have been read in within the last five minutes. You can use ListHDFS or GenerateFlowFile to create the flowfiles that this processor expects.

如果这不是问题，您还可以检查处理器调度以确保它设置为频繁运行(它应该是事件驱动的，但是如果由于某种原因设置了计时器并且很长，这可能会导致这种行为).

If that’s not the issue, you can also check the processor schedule to make sure it is set to run frequently (it should be event driven, but if the timer is set and long for some reason, that could be causing this behavior).

至于新的 HDFS 文件系统，您是否在MoveHDFS 处理器属性?如果找不到现有文件，它将创建默认文件.Apache NiFi 站点.

As for the new HDFS file system, did you provide the (correct and fully-pathed) core-site.xml and hdfs-site.xml files in the MoveHDFS processor properties? If existing files are not found, it will create default ones. There is full documentation available at the Apache NiFi site.

这篇关于NiFi moveHDFS 处理器似乎什么都不做的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！