如何在不使用 Web 仪表板的情况下分析 Apache Storm 拓扑? [英] How can I profile Apache Storm topologies without using the web dashboard?

问题描述

标题几乎说明了一切，我有一些 Storm 拓扑，我想测量它们的延迟，即从 Kafka 传入消息到最终相关执行的最后一点之间的时间量螺栓.如果我可以深入研究结果以查看每个螺栓的延迟，则加分.

The title pretty much says it all, I have some Storm topologies and I'd like to measure their latencies, that is, the amount of time between a message coming in from Kafka and the last bit of related execution in the final bolt. Bonus points if I can drill into the results to see the latency across each bolt.

这可以通过简单地调整 Storm 配置来完成吗?

Can this be done by simply tweaking the Storm configuration?

如果没有，是 http://storm.incubator.apache.org/apidocs/backtype/storm/hooks/info/SpoutAckInfo.html backtype.storm.hooks.info.SpoutAckInfo.completeLatencyMs 我在找什么?

If not, is http://storm.incubator.apache.org/apidocs/backtype/storm/hooks/info/SpoutAckInfo.html backtype.storm.hooks.info.SpoutAckInfo.completeLatencyMs the thing I'm looking for?

推荐答案

对于一般分析，我们有一个 fork of Storm，我允许主管动态设置和打开等于 1000 的 JMX 端口，比工作端口(的当然，这限制了我们可以分配的工人数量，但我们现在离这个问题还很远).然后使用 supervisor.childopts 值分配工作线程的 JMX 端口并连接到 Visual VM.

For general profiling, we have a fork of storm where I enable the supervisor to dynamically set and open the JMX port equal to 1000 higher than the worker port (of course, this limits the number of workers we could assign, but we're nowhere near that problem now). Then with the supervisor.childopts value we assign the worker's JMX port and connect with Visual VM.

具体到您的指标问题，您可以使用 BaseTaskHook 将风暴指标推送到带有 MBean(我们使用 codahale 指标)的东西，并再次通过 JMX 浏览值.我发布了一个示例 gist.请注意，我们尚未将 codahale 指标提升至 3.x.

Specific to your metrics question, you can use the BaseTaskHook to push the storm metrics to something with an MBean (we use codahale metrics) and again, browse the values over JMX. I posted an example gist. Note we haven't uplifted to 3.x of codahale metrics.

使用 Java 7 工具，我相信您可以在正在运行的 JVM 上打开 JMX，所以如果您使用的是 Java 7+，那也会对您有所帮助.

With Java 7 tools, I believe you're able to turn on JMX on a running JVM, so if you're on Java 7+, that would also help you.

这篇关于如何在不使用 Web 仪表板的情况下分析 Apache Storm 拓扑?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！