使用来自多个 kafka 主题的消息的最佳实践是什么? [英] what is best practice to consume messages from multiple kafka topics?
问题描述
我需要消费来自不同 kafka 主题的消息，
I need to consumer messages from different kafka topics,
- 我是否应该为每个主题创建不同的消费者实例，然后根据分区数量启动一个新的处理线程.或
- 我应该从单个消费者实例订阅所有主题，并且应该启动不同的处理线程
谢谢&问候，梅加
Thanks & regards, Megha
推荐答案
唯一的规则是你必须考虑 Kafka 做什么和不保证什么:
The only rule is that you have to account for what Kafka does and doesn't not guarantee:
- Kafka 只保证单个主题/分区的消息顺序.这也意味着如果您的单个主题消费者出于某种原因切换分区，您可能会无序获取消息.
- 当您使用单个消费者订阅多个主题时，会为每个请求的主题为该消费者分配一个主题/分区对.
- 这意味着任何一个主题的传入消息的顺序都是正确的，但您不能保证主题之间的顺序是按时间顺序排列的.
- 您也不能保证在任何给定的时间段内都会收到来自任何特定订阅主题的消息.
我最近遇到了一个错误，因为我的应用程序通过一个消费者订阅了许多主题.每个主题都是一个实时图像源，每条消息一个图像.由于所有主题总是有新图像，因此每次 poll() 仅返回要注册的第一个主题的图像.
I recently had a bug because my application subscribed to many topics with a single Consumer. Each topic was a live feed of images at one image per message. Since all the topics always had new images, each poll() was only returning images from the first topic to register.
如果处理所有消息很重要，您需要确保每个消费者可以比创建消息更快地处理来自其所有订阅主题的消息.如果不能，您要么需要更多消费者在同一组中提交读取，要么您必须接受某些消息可能永远不会被处理的事实.
If processing all messages is important, you'll need to be certain that each Consumer can process messages from all of its subscribed topics faster than the messages are created. If it can't, you'll either need more Consumers committing reads in the same group, or you'll have to be OK with the fact that some messages may never be processed.
显然，一个消费者/主题是最简单的，但它确实增加了一些额外的消费者的开销.您必须根据自己的需要确定这是否重要.
Obviously one Consumer/topic is the simplest, but it does add some overhead to have the additional Consumers. You'll have to determine whether that's important based on your needs.
正确回答您的问题的唯一方法是评估您的应用程序的特定要求和功能，并构建在这些和在 Kafka 的限制范围内工作的东西.
The only way to correctly answer your question is to evaluate your application's specific requirements and capabilities, and build something that works within those and within Kafka's limitations.
这篇关于使用来自多个 kafka 主题的消息的最佳实践是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
