使用 2 个消费者配置时消费者吞吐量缓慢 [英] Slow consumer throuput when using 2 consumer-configuration
问题描述
使用 spring-integration-kafka 扩展和以下配置:
Using the spring-integration-kafka extention and the following configuration:
<int-kafka:zookeeper-connect id="zookeeperConnect"
zk-connect="#{kafkaConfig['zooKeeperUrl']}" zk-connection-timeout="10000"
zk-session-timeout="10000" zk-sync-time="2000" />
<int-kafka:consumer-context id="consumerContext" consumer-timeout="5000" zookeeper-connect="zookeeperConnect">
<int-kafka:consumer-configurations>
<int-kafka:consumer-configuration
group-id="realtime-services-consumer-grp"
value-decoder="purchaseDecoder"
key-decoder="kafkaReflectionDecoder"
max-messages="5" >
<int-kafka:topic id="purchase" streams="1" />
</int-kafka:consumer-configuration>
<int-kafka:consumer-configuration
group-id="realtime-services-consumer-gw"
value-decoder="eventDecoder"
key-decoder="kafkaReflectionDecoder"
max-messages="10" >
<int-kafka:topic id="event" streams="1" />
</int-kafka:consumer-configuration>
</int-kafka:consumer-configurations>
</int-kafka:consumer-context>
<int-kafka:inbound-channel-adapter
id="kafkaInboundChannelAdapter" kafka-consumer-context-ref="consumerContext"
auto-startup="true" channel="inputFromKafka">
<int:poller fixed-delay="20" time-unit="MILLISECONDS" />
</int-kafka:inbound-channel-adapter>
例如,当我评论第一个 consumer-configuration
时,我每分钟可以有 300 个事件而没有问题.但是当两者都被激活时.我的吞吐量非常低.来自这两个主题的总吞吐量低于每分钟 50.
When for example I comment the first consumer-configuration
I can have 300 event per minute with no issue. But when both are activated. I have a very low throughput. Total of throughput coming from both topic is less than a 50 per minutes.
有谁知道为什么我在阅读 2 个主题时表现如此糟糕?我在配置中做错了什么?
Anyone know why I have such poor performance when reading from 2 topic? What did I do wrong in the configuration?
推荐答案
感谢您指出这一点!
在与我当地的 Kafka claster 进行一番激烈斗争后,我已经能够重现您的问题,并且我为您提供了一些解决方法:-).
After some big fight with my local Kafka claster I have been able to reproduce your issue and I am with some workaround for you :-).
首先不是round-robin
,而是一个接一个:
First of all it isn't round-robin
, but one by one:
for (final ConsumerConfiguration<K, V> consumerConfiguration : getConsumerConfigurations().values()) {
Map<String, Map<Integer, List<Object>>> messages = consumerConfiguration.receive();
如果 KafkaStream
中没有消息,在 consumer-timeout="5000"
期间,每个 consumerConfiguration
在后台被阻止代码>现在.因此,来自 <int-kafka:inbound-channel-adapter>
的整个 poll
任务被阻塞,直到超时或更糟:如果每个主题没有消息整个等待超时是超时的总和!
Where each of those consumerConfiguration
are blocked on the background during that consumer-timeout="5000"
, if there is no messages in the KafkaStream
right now. Hence entire poll
task from the <int-kafka:inbound-channel-adapter>
is blocked until that timeout or even worse: if each topic doesn't have message the whole waiting time out is sum of timeouts!
要解决这个问题,您可以减少 consumer-timeout="5000"
或提供几个
和
用于每个主题.
To overcome the issue, you can decrease consumer-timeout="5000"
or provide several <int-kafka:consumer-context>
and therefore <int-kafka:inbound-channel-adapter>
for each topic.
是的,这看起来很奇怪,而且在发布之前我们还没有找到时间来看看这真的很糟糕,但无论如何请随时提出 JIRA 问题来修复它.
Yes, it looks odd and it is really bad that we haven't found a time to take a look to this before release, but anyway feel free to raise JIRA issue to fix it.
谢谢!
这篇关于使用 2 个消费者配置时消费者吞吐量缓慢的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!