ack引起的风暴延迟 [英] Storm latency caused by ack

问题描述

我正在使用kafka-storm连接kafka和Storm.我有3台运行Zookeeper，Kafka和Storm的服务器. kafka中有一个主题"test"，其中有9个分区.

I was using kafka-storm to connect kafka and storm. I have 3 servers running zookeeper, kafka and storm. There is a topic 'test' in kafka that has 9 partitions.

在风暴拓扑中，KafkaSpout执行器的数量为9，默认情况下，任务的数量也应为9.而提取"螺栓是唯一连接到对数"喷嘴KafkaSpout的螺栓.

In the storm topology, the number of KafkaSpout executor is 9 and by default, the number of tasks should be 9 as well. And the 'extract' bolt is the only bolt connected to KafkaSpout, the 'log' spout.

在用户界面中，喷口失败率很高.但是，螺栓中执行的消息数=发出的消息数-螺栓中失败的消息数.当失败的消息在开始时为空时，此方程几乎匹配.

From the UI, there is a huge rate of failure in the spout. However, he number of executed message in bolt = the number of emitted message - the number of failed mesage in bolt. This equation is almost matched when the failed message is empty at the beginning.

根据我的理解，这意味着螺栓确实收到了来自水嘴的消息，但ack信号在飞行中被挂起.这就是壶嘴中的袜子如此之少的原因.

Based on my understanding, this means that the bolt did receive the message from spout but the ack signals are suspended in flight. That's the reason why the number of acks in spout are so small.

可以通过增加超时秒数和吐出未决消息数来解决此问题.但这将导致更多的内存使用，并且我无法将其增加到无限.

This problem might be solved by increase the timeout seconds and spout pending message number. But this will cause more memory usage and I cannot increase it to infinite.

我在徘徊，是否有办法强制风暴忽略某些喷口/螺栓上的确认，这样它才在超时之前不会等待该信号.这应该显着增加整个过程，但不能保证消息处理.

I was wandering if there is a way to force storm ignore the ack in some spout/bolt, so that it will not waiting for that signal until time out. This should increase the throughout significantly but not guarantee for message processing.

推荐答案

如果将ackers的数量设置为0，则Storm将自动确认每个样本.

if you set the number of ackers to 0 then storm will automatically ack every sample.

config.setNumAckers(0);

请注意，UI仅测量并显示5％的数据流. 除非您设置

please note that the UI only measures and shows 5% of the data flow. unless you set

config.setStatsSampleRate(1.0d);

尝试增加销钉的超时时间并减少topology.max.spout.pending的数量.

try increasing the bolt's timeout and reducing the amount of topology.max.spout.pending.

还要确保喷嘴的nextTuple()方法是非阻塞的且已优化.

also, make sure the spout's nextTuple() method is non blocking and optimized.

我还建议对代码进行概要分析，也许您的风暴队列已满，并且您需要增加它们的大小.

i would also recommend profiling the code, maybe your storm Queues are being filled and you need to increase their sizes.

    config.put(Config.TOPOLOGY_TRANSFER_BUFFER_SIZE,32);
    config.put(Config.TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE,16384);
    config.put(Config.TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE,16384);

这篇关于ack引起的风暴延迟的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！