Apache Storm:指标日志文件为空 [英] Apache Storm : Metrics log file is empty

问题描述

我正在尝试遵循此处的示例

I am trying to follow the example here

https://www.endgame.com/blog/storm-metrics-how

这是我的storm.yaml

here is my storm.yaml

storm.zookeeper.servers:
- localhost


supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704


nimbus.host: localhost

ui.port: 8080
ui.host: localhost

storm.log.dir: /path/to/storm/logdir

topology.max.spout.pending: 5000

我尝试在本地和群集模式下运行拓扑.该metrics.log文件在/path/to/storm/logdir位置创建，但是该文件为空！我缺少一些配置吗?

I tried running the topology in local and cluster mode. the metrics.log file is created at the location /path/to/storm/logdir but the file is empty! am i missing some configuration?

推荐答案

问题与 Storm中的指标的当前 log4j2 设置有关，该修复程序涉及一些问题

The problem is with the current log4j2 setup of Metrics in Storm and the fix is a little involved.

首先，让我列出一些来自Storm的Jira的错误.阅读这些问题和修复方法可以通过度量标准日志记录来很好地了解手头的问题:

Firstly let me list some bugs from Storm's Jira. Reading the issues and fixes sheds a good amount of light onto the issue at hand with metrics logging:

• STORM-584整个集群正在共享指标日志
• LoggingMetricsConsumer缺少指标记录器设置的 STORM-1673包重命名
• STORM-1767使用完全限定的类名命名记录器会产生问题

• STORM-584 whole cluster is sharing the metrics log
• STORM-1673 package rename of LoggingMetricsConsumermissed metrics logger setup
• STORM-1767 naming logger using fully qualified class name creates problems

总结以上情况:

• 在cluster.xml中定义指标记录器是不正确的，因为使用worker.xml定义了工作记录器(记录了记录的记录).因此，工作人员可能无法访问指标记录器定义.它还会带来文件所有权的问题-第一个记录指标的工作人员将拥有该日志文件，从而阻止其他工作人员中的其他记录程序附加到指标日志文件中.
• 将包从backtype.storm.metric重命名为org.apache.storm.metric后，该包将重命名丢失的worker.xml文件.
• 最后，指标记录器被命名为org.apache.storm.metric.LoggingMetricsConsumer，这意味着当尝试使用LoggingMetricsConsumer类中的LoggerFactory.getLogger(LoggingMetricsConsumer.class)创建它时，实际创建的记录器是连接到 ROOT 的记录器.记录器，并使用A1附加程序而不是METRICS附加程序.

• Defining metrics logger in cluster.xml is not right as worker loggers (where metrics are being logged from) are defined using worker.xml. So it's possible that a worker will not have access to the metrics logger definition. It also creates problems of file ownership - the first worker to log metrics would own the log file preventing other loggers in other workers from appending to metrics log file.
• After the rename of the package from backtype.storm.metric to org.apache.storm.metric the package rename missed worker.xml file.
• Finally the metrics logger is named org.apache.storm.metric.LoggingMetricsConsumer which means that when trying to create it using LoggerFactory.getLogger(LoggingMetricsConsumer.class) inside the LoggingMetricsConsumer class, the logger that is actually created is a logger attached to ROOT logger and using A1 appender instead of METRICS appender.

对我来说，实际修复涉及两件事(请注意，第一步是可选的)

The actual fix for me involved 2 things (please note that first step is optional)

1. (可选)整理worker.xml和cluster.xml文件中的 log4j2 设置.
   只需确保您的日志记录设置具有worker.xml中定义的指标记录器，就像在 github中定义的一样提交给Storm项目，以解决STORM-1673问题(已导致1.0.0和2.0.0).相反，请确保不再在cluster.xml文件中定义指标记录器.

1. Optionally sorting out log4j2 setup in worker.xml and cluster.xml files.
   Simply make sure your logging setup has the metrics logger defined in worker.xml like it is in this github commit for Storm project fixing the STORM-1673 issue (which has made it to 1.0.0 and 2.0.0). Conversely please also make sure that the metrics logger is no longer defined in the cluster.xml file anymore.

重命名指标记录器并派发LoggingMetricsConsumer类将其指向新重命名的记录器(未分配给 root 记录器):

首先，请确保您在worker.xml文件中的指标记录器不再命名为org.apache.storm.metric.LoggingMetricsConsumer，而是按照以下代码片段说出METRICS_LOGGER:

Renaming the metrics logger and forking LoggingMetricsConsumer class to point it at a newly renamed logger (unatached to root logger):

Firstly make sure your metrics logger in worker.xml file is no longer named org.apache.storm.metric.LoggingMetricsConsumer but say METRICS_LOGGER as per this snippet:

<Logger name="METRICS_LOGGER" level="info" additivity="false">
    <appender-ref ref="METRICS"/>
</Logger>

其次来自同一

Secondly "fork" LoggingMetricsConsumer class from the same github commit as above.
Please also note that by fork I mean whatever it takes to bring its source code into your project in order to modify it. You can also define your own class implementing IMetricsConsumer as long as you then attach it to your topology. Once you have your own LoggingMetricsConsumer class you have to modify the way it creates its metrics logger to:

public static final Logger METRICS_LOG = LoggerFactory.getLogger("METRICS_LOGGER");

请注意，如果您使用的Storm版本不同于1.0.0或2.0.0

只需确保您查看了LoggingMetricsConsumer源代码以及git分支中worker.xml和cluster.xml的版本，这些版本已被剪切掉.该区域已经发生了很大变化，并且每个变化都不是向后兼容的(软件包重命名，记录器移动).

Last note if you are using version of Storm different than 1.0.0 or 2.0.0

Just make sure you look at the LoggingMetricsConsumer source code and the versions of worker.xml and cluster.xml from the git branch your version has been cut from. This area has changed a lot and each change is not backwards compatible (package rename, logger move).

这篇关于Apache Storm:指标日志文件为空的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！