Apache Storm:指标日志文件为空 [英] Apache Storm : Metrics log file is empty
问题描述
我正在尝试遵循此处的示例
I am trying to follow the example here
https://www.endgame.com/blog/storm-metrics-how
这是我的storm.yaml
here is my storm.yaml
storm.zookeeper.servers:
- localhost
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704
nimbus.host: localhost
ui.port: 8080
ui.host: localhost
storm.log.dir: /path/to/storm/logdir
topology.max.spout.pending: 5000
我尝试在本地和集群模式下运行拓扑.metrics.log 文件创建在位置 /path/to/storm/logdir
但该文件是空的!我缺少一些配置吗?
I tried running the topology in local and cluster mode. the metrics.log file is created at the location /path/to/storm/logdir
but the file is empty! am i missing some configuration?
推荐答案
问题在于当前 log4j2 Storm 中的度量 设置,修复有点复杂.
The problem is with the current log4j2 setup of Metrics in Storm and the fix is a little involved.
首先让我列出 Storm 的 Jira 中的一些错误.通过指标记录,阅读问题和修复可以很好地了解手头的问题:
Firstly let me list some bugs from Storm's Jira. Reading the issues and fixes sheds a good amount of light onto the issue at hand with metrics logging:
STORM-584
whole cluster is sharing the metrics logSTORM-1673
package rename ofLoggingMetricsConsumer
missed metrics logger setupSTORM-1767
naming logger using fully qualified class name creates problems
总结以上情况:
- 在
cluster.xml
中定义 metrics 记录器是不正确的,因为工作器记录器(记录指标的位置)是使用worker.xml
.因此,工作人员可能无法访问指标记录器定义.它还造成文件所有权问题 - 记录指标的第一个工作人员将拥有日志文件,从而防止其他工作人员中的其他记录员附加到指标日志文件. - 将包从
backtype.storm.metric
重命名为org.apache.storm.metric
后,包重命名丢失了worker.xml
文件. - 最后,度量记录器被命名为
org.apache.storm.metric.LoggingMetricsConsumer
这意味着当尝试使用LoggerFactory.getLogger(LoggingMetricsConsumer.class)
创建它时LoggingMetricsConsumer
类,实际创建的记录器是附加到 ROOT 记录器并使用A1
appender 而不是METRICS代码> appender.
- Defining metrics logger in
cluster.xml
is not right as worker loggers (where metrics are being logged from) are defined usingworker.xml
. So it's possible that a worker will not have access to the metrics logger definition. It also creates problems of file ownership - the first worker to log metrics would own the log file preventing other loggers in other workers from appending to metrics log file. - After the rename of the package from
backtype.storm.metric
toorg.apache.storm.metric
the package rename missedworker.xml
file. - Finally the metrics logger is named
org.apache.storm.metric.LoggingMetricsConsumer
which means that when trying to create it usingLoggerFactory.getLogger(LoggingMetricsConsumer.class)
inside theLoggingMetricsConsumer
class, the logger that is actually created is a logger attached to ROOT logger and usingA1
appender instead ofMETRICS
appender.
对我来说实际的修复涉及两件事(请注意,第一步是可选的)
The actual fix for me involved 2 things (please note that first step is optional)
可选择整理
worker.xml
和cluster.xml
文件中的 log4j2 设置.
只需确保您的日志记录设置具有在worker.xml
中定义的指标记录器,就像在此 github 提交修复了STORM-1673
问题(已使其成为1.0.0
和2.0.0
>).相反,请确保不再在cluster.xml
文件中定义指标记录器.
Optionally sorting out log4j2 setup in
worker.xml
andcluster.xml
files.
Simply make sure your logging setup has the metrics logger defined inworker.xml
like it is in this github commit for Storm project fixing theSTORM-1673
issue (which has made it to1.0.0
and2.0.0
). Conversely please also make sure that the metrics logger is no longer defined in thecluster.xml
file anymore.
重命名指标记录器并分叉 LoggingMetricsConsumer
类以将其指向新重命名的记录器(未附加到 root 记录器):
首先确保 worker.xml
文件中的指标记录器不再命名为 org.apache.storm.metric.LoggingMetricsConsumer
而是说 METRICS_LOGGER
根据此代码段:
Renaming the metrics logger and forking LoggingMetricsConsumer
class to point it at a newly renamed logger (unatached to root logger):
Firstly make sure your metrics logger in worker.xml
file is no longer named org.apache.storm.metric.LoggingMetricsConsumer
but say METRICS_LOGGER
as per this snippet:
<Logger name="METRICS_LOGGER" level="info" additivity="false">
<appender-ref ref="METRICS"/>
</Logger>
其次fork"LoggingMetricsConsumer
类来自同一个 github 提交如上.
另请注意fork 我的意思是将其源代码带入您的项目以对其进行修改所需的一切.你也可以定义你自己的实现 IMetricsConsumer
的类,只要你把它附加到你的拓扑中.一旦你有了自己的 LoggingMetricsConsumer
类,你必须修改它创建度量记录器的方式:
Secondly "fork" LoggingMetricsConsumer
class from the same github commit as above.
Please also note that by fork I mean whatever it takes to bring its source code into your project in order to modify it. You can also define your own class implementing IMetricsConsumer
as long as you then attach it to your topology. Once you have your own LoggingMetricsConsumer
class you have to modify the way it creates its metrics logger to:
public static final Logger METRICS_LOG = LoggerFactory.getLogger("METRICS_LOGGER");
如果您使用的 Storm 版本不是 1.0.0 或 2.0.0,最后请注意
请确保您查看了 LoggingMetricsConsumer
源代码以及您的 git 分支中的 worker.xml
和 cluster.xml
版本版本已被删减.这个区域已经发生了很大的变化,而且每次变化都不是向后兼容的(包重命名,记录器移动).
Last note if you are using version of Storm different than 1.0.0 or 2.0.0
Just make sure you look at the LoggingMetricsConsumer
source code and the versions of worker.xml
and cluster.xml
from the git branch your version has been cut from. This area has changed a lot and each change is not backwards compatible (package rename, logger move).
这篇关于Apache Storm:指标日志文件为空的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!