吸收来自多个kafka主题的消息的最佳实践是什么? [英] what is best practice to consume messages from multiple kafka topics?

问题描述

我需要吸收来自不同kafka主题的消息，

I need to consumer messages from different kafka topics,

1. 我应该为每个主题创建不同的使用者实例，然后根据分区数启动一个新的处理线程. 或
2. 我应该订阅一个消费者实例中的所有主题，并且应该启动不同的处理线程

1. Should i create different consumer instance per topic and then start a new processing thread as per the number of partition. or
2. I should subscribe all topics from a single consumer instance and the should start different processing threads

感谢&问候， 梅加

Thanks & regards, Megha

推荐答案

唯一的规则是您必须考虑Kafka所做的事情，但不能保证:

The only rule is that you have to account for what Kafka does and doesn't not guarantee:

• Kafka仅保证单个主题/分区的消息顺序.这也意味着，如果您的单个主题使用者"出于某些原因切换了分区，则可能会使消息混乱.
• 当您通过一个使用者订阅多个主题时，该消费者将为每个请求的主题分配一个主题/分区对.
• 这意味着任何一个主题的传入消息顺序都是正确的，但是您不能保证主题之间的顺序是按时间顺序排列的.
• 您也不能保证在任何给定的时间段内都会收到来自任何特定订阅主题的消息.

我最近遇到了一个错误，因为我的应用程序通过一个Consumer订阅了许多主题.每个主题都是图像的实时供稿，每条消息一个图像.由于所有主题始终都有新图像，因此每个poll()仅从第一个主题返回图像进行注册.

I recently had a bug because my application subscribed to many topics with a single Consumer. Each topic was a live feed of images at one image per message. Since all the topics always had new images, each poll() was only returning images from the first topic to register.

如果处理所有消息很重要，则需要确保每个使用者都能比其创建消息时更快地处理其所有订阅主题中的消息.如果不能，那么您要么需要更多的使用者在同一组中提交读取，要么必须确定某些消息可能永远不会被处理，这一点您可以确定.

If processing all messages is important, you'll need to be certain that each Consumer can process messages from all of its subscribed topics faster than the messages are created. If it can't, you'll either need more Consumers committing reads in the same group, or you'll have to be OK with the fact that some messages may never be processed.

很明显，一个消费者/主题"是最简单的，但确实增加了一些额外的消费者的开销.您必须根据自己的需求确定这是否重要.

Obviously one Consumer/topic is the simplest, but it does add some overhead to have the additional Consumers. You'll have to determine whether that's important based on your needs.

正确回答问题的唯一方法是评估应用程序的特定要求和功能，并在卡夫卡(Kafka)的限制范围内构建可在这些和中使用的功能.

The only way to correctly answer your question is to evaluate your application's specific requirements and capabilities, and build something that works within those and within Kafka's limitations.

这篇关于吸收来自多个kafka主题的消息的最佳实践是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！