NiFi moveHDFS处理器似乎什么都不做 [英] NiFi moveHDFS processor appears to do nothing
问题描述
处理器指标在长时间后保持为零
并且在公告板上显示没有错误（日志记录级别设置为INFO），公告板中的唯一日志记录输出为：
14:50:04 HSTINFO1e637d0d-0163-1000-7bde-a7993ae403e8
MoveHDFS [id = 1e637d0d-0163-1000-7bde-a7993ae403e8]已初始化一个带有工作目录的新HDFS文件系统：file：/home/mapr/nifi-1.6.0默认块大小：33554432默认复制：1 config：配置：core-default.xml，core-site.xml，mapred-default。 xml，mapred-site.xml，yarn-default.xml，yarn-site.xml，hdfs-default.xml，hdfs-site.xml，/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core -site.xml，/opt/mapr/hadoop/hadoop-2.7.0 /etc/hadoop/hdfs-site.xml
<顺便说一句，为什么会说Initialized一个新的HDFS文件系统？它不应该使用提供的hadoop配置文件来访问hadoop集群上现有的HDFS吗？）。将NiFi作为单个实例安装在hadoop群集节点上，并根据需要将core-和hdfs-site.xml文件添加到处理器属性中。以前从未使用过NiFi，并怀疑我错过了在这里正确配置流程的明显问题。任何意见关于这里发生了什么，以及如何解决它将不胜感激。谢谢。
如果您可以分享更多流量，那将会很有帮助。 MoveHDFS 处理器需要传入的流文件才能处理它们，因此生成或接收流文件的任何处理器都需要将它们传递到此处理器。队列是否显示任何排队的流程文件？处理器显示在最近五分钟内没有读取流文件。您可以使用 ListHDFS 或 GenerateFlowFile 创建此处理器所需的流文件。如果这不是问题，您还可以检查处理器时间表，确保它被设置为频繁运行（它应该是事件驱动的，但是如果设置了定时器并且由于某种原因而长期可能会导致这种行为）。
至于新的HDFS文件系统，您是否提供了（正确和完整路径） core-site.xml 和 hdfs-site.xml 文件位于 MoveHDFS 处理器属性中？如果找不到现有文件，它将创建默认文件。在 Apache NiFi网站。
Simply trying to automate a move of files from one HDFS dir to another. Doing this with a moveHDFS processor in Apache NiFi, but when starting the processor nothing seems to happen.
The processor metrics remain at zero after long amount of time and looking at the bulletin board shows no errors (logging level set to INFO), the only logging output in the bulletin board is:
14:50:04 HSTINFO1e637d0d-0163-1000-7bde-a7993ae403e8
MoveHDFS[id=1e637d0d-0163-1000-7bde-a7993ae403e8] Initialized a new HDFS File System with working dir: file:/home/mapr/nifi-1.6.0 default block size: 33554432 default replication: 1 config: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml, /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/hdfs-site.xml
(By the way, why does it say "Initialized a new HDFS File System"? Shouldn't it be using the provided hadoop config files to access the existing HDFS on the hadoop cluster?). Have NiFi installed as a single instance on a hadoop cluster node and added the core- and hdfs-site.xml files to the processor properties as needed. Have never used NiFi before and suspect that I am missing something obvious about properly configuring the flow here. Any advice about what is going on here and how to fix it would be appreciated. Thanks.
If you can share more of your flow, that would be helpful. The MoveHDFS processor requires incoming flowfiles in order to process them, so whatever processor is generating or receiving flowfiles needs to pass them on to this processor. Does the queue show any queued flowfiles? The processor shows that no flowfiles have been read in within the last five minutes. You can use ListHDFS or GenerateFlowFile to create the flowfiles that this processor expects.
If that’s not the issue, you can also check the processor schedule to make sure it is set to run frequently (it should be event driven, but if the timer is set and long for some reason, that could be causing this behavior).
As for the new HDFS file system, did you provide the (correct and fully-pathed) core-site.xml and hdfs-site.xml files in the MoveHDFS processor properties? If existing files are not found, it will create default ones. There is full documentation available at the Apache NiFi site.
这篇关于NiFi moveHDFS处理器似乎什么都不做的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
