Apache Storm:指标日志文件为空 [英] Apache Storm : Metrics log file is empty

问题描述

我正在尝试遵循此处的示例

I am trying to follow the example here

https://www.endgame.com/blog/storm-metrics-how

这是我的storm.yaml

here is my storm.yaml

storm.zookeeper.servers:
- localhost


supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704


nimbus.host: localhost

ui.port: 8080
ui.host: localhost

storm.log.dir: /path/to/storm/logdir

topology.max.spout.pending: 5000

我尝试在本地和集群模式下运行拓扑.metrics.log 文件创建在位置 /path/to/storm/logdir 但该文件是空的！我缺少一些配置吗?

I tried running the topology in local and cluster mode. the metrics.log file is created at the location /path/to/storm/logdir but the file is empty! am i missing some configuration?

推荐答案

问题在于当前 log4j2 Storm 中的度量 设置，修复有点复杂.

The problem is with the current log4j2 setup of Metrics in Storm and the fix is a little involved.

首先让我列出 Storm 的 Jira 中的一些错误.通过指标记录，阅读问题和修复可以很好地了解手头的问题:

Firstly let me list some bugs from Storm's Jira. Reading the issues and fixes sheds a good amount of light onto the issue at hand with metrics logging:

• STORM-584 整个集群正在共享指标日志
• STORM-1673 包重命名 LoggingMetricsConsumer错过的指标记录器设置
• STORM-1767 使用完全限定的类名命名记录器创建问题

• STORM-584 whole cluster is sharing the metrics log
• STORM-1673 package rename of LoggingMetricsConsumermissed metrics logger setup
• STORM-1767 naming logger using fully qualified class name creates problems

总结以上情况:

• 在 cluster.xml 中定义 metrics 记录器是不正确的，因为工作器记录器(记录指标的位置)是使用 worker.xml.因此，工作人员可能无法访问指标记录器定义.它还造成文件所有权问题 - 记录指标的第一个工作人员将拥有日志文件，从而防止其他工作人员中的其他记录员附加到指标日志文件.
• 将包从 backtype.storm.metric 重命名为 org.apache.storm.metric 后，包重命名丢失了 worker.xml 文件.
• 最后，度量记录器被命名为 org.apache.storm.metric.LoggingMetricsConsumer 这意味着当尝试使用 LoggerFactory.getLogger(LoggingMetricsConsumer.class) 创建它时LoggingMetricsConsumer 类，实际创建的记录器是附加到 ROOT 记录器并使用 A1 appender 而不是 METRICS appender.

• Defining metrics logger in cluster.xml is not right as worker loggers (where metrics are being logged from) are defined using worker.xml. So it's possible that a worker will not have access to the metrics logger definition. It also creates problems of file ownership - the first worker to log metrics would own the log file preventing other loggers in other workers from appending to metrics log file.
• After the rename of the package from backtype.storm.metric to org.apache.storm.metric the package rename missed worker.xml file.
• Finally the metrics logger is named org.apache.storm.metric.LoggingMetricsConsumer which means that when trying to create it using LoggerFactory.getLogger(LoggingMetricsConsumer.class) inside the LoggingMetricsConsumer class, the logger that is actually created is a logger attached to ROOT logger and using A1 appender instead of METRICS appender.

对我来说实际的修复涉及两件事(请注意，第一步是可选的)

The actual fix for me involved 2 things (please note that first step is optional)

1. 可选择整理 worker.xml 和 cluster.xml 文件中的 log4j2 设置.
   只需确保您的日志记录设置具有在 worker.xml 中定义的指标记录器，就像在此 github 提交修复了 STORM-1673 问题(已使其成为 1.0.0 和 2.0.0>).相反，请确保不再在 cluster.xml 文件中定义指标记录器.

1. Optionally sorting out log4j2 setup in worker.xml and cluster.xml files.
   Simply make sure your logging setup has the metrics logger defined in worker.xml like it is in this github commit for Storm project fixing the STORM-1673 issue (which has made it to 1.0.0 and 2.0.0). Conversely please also make sure that the metrics logger is no longer defined in the cluster.xml file anymore.

重命名指标记录器并分叉 LoggingMetricsConsumer 类以将其指向新重命名的记录器(未附加到 root 记录器):

首先确保 worker.xml 文件中的指标记录器不再命名为 org.apache.storm.metric.LoggingMetricsConsumer 而是说 METRICS_LOGGER 根据此代码段:

Renaming the metrics logger and forking LoggingMetricsConsumer class to point it at a newly renamed logger (unatached to root logger):

Firstly make sure your metrics logger in worker.xml file is no longer named org.apache.storm.metric.LoggingMetricsConsumer but say METRICS_LOGGER as per this snippet:

<Logger name="METRICS_LOGGER" level="info" additivity="false">
    <appender-ref ref="METRICS"/>
</Logger>

其次fork"LoggingMetricsConsumer 类来自同一个 github 提交如上.
另请注意fork 我的意思是将其源代码带入您的项目以对其进行修改所需的一切.你也可以定义你自己的实现 IMetricsConsumer 的类，只要你把它附加到你的拓扑中.一旦你有了自己的 LoggingMetricsConsumer 类，你必须修改它创建度量记录器的方式:

Secondly "fork" LoggingMetricsConsumer class from the same github commit as above.
Please also note that by fork I mean whatever it takes to bring its source code into your project in order to modify it. You can also define your own class implementing IMetricsConsumer as long as you then attach it to your topology. Once you have your own LoggingMetricsConsumer class you have to modify the way it creates its metrics logger to:

public static final Logger METRICS_LOG = LoggerFactory.getLogger("METRICS_LOGGER");

如果您使用的 Storm 版本不是 1.0.0 或 2.0.0，最后请注意

请确保您查看了 LoggingMetricsConsumer 源代码以及您的 git 分支中的 worker.xml 和 cluster.xml 版本版本已被删减.这个区域已经发生了很大的变化，而且每次变化都不是向后兼容的(包重命名，记录器移动).

Last note if you are using version of Storm different than 1.0.0 or 2.0.0

Just make sure you look at the LoggingMetricsConsumer source code and the versions of worker.xml and cluster.xml from the git branch your version has been cut from. This area has changed a lot and each change is not backwards compatible (package rename, logger move).

这篇关于Apache Storm:指标日志文件为空的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！