Kafka如何并行消费一个主题 [英] Kafka how to consume one topic parallel

问题描述

看了kafka文档，还是不知道如何并行消费一个topic?

I read kafka document, still don't know how consume one topic parallel?

假设:我有一个像发生了什么事"这样的话题(不要拆分这个话题)，我有很多客户想要消费它.那么我该怎么做，让多个客户可以并行消费呢?我应该使用分区和客户组吗?

Suppose: I have one topic like "something happened" (don't split this topic), and I have many customers that want to consume it. So what should I do, so that multiple customers can consume it parallel? Should I use partitioning and customer groups?

我对此有一个想法，但我不确定它是否正确.

I have one idea about this, but I'm not sure whether is it right.

对同一个topic做很多partition，对一个customer做一个partition，那么一个producer必须对这些partition做同样的事情，每个customer都在不同的customer group里，对不对?

Make many partitions about the same topic, and make one partition to one customer, so one producer must produce the same to these partitions, and every customer in the different customer group, is it right?

推荐答案

使用分区是能够并行化主题消费的方式.假设您的主题有 10 个分区，那么同一个消费者组中可以有 10 个消费者，每个消费者读取一个分区.如果您的消费者少于分区，那么他们每个人将负责一个以上的分区.如果您的消费者比分区多，那么就会有消费者不会获得任何分配给他们的分区，除了可以替换另一个已经死亡的消费者之外，无事可做.

Using partitions is the way of being able to parallelize the consumption of a topic. Let´s say you have 10 partitions for your topic, then you can have 10 consumers in the same consumer group reading one partition each. If you have less consumers than partitions, then they would be responsible for more than one partition each. If you have more consumers than partitions, then there would be consumers who would not get any partition assigned to them and have nothing to do except being available to replace another consumer who has died.

这篇关于Kafka如何并行消费一个主题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！