卡夫卡的平行生产和消费 [英] Parallel Producing and Consuming in Kafka

问题描述

1.同时使用同一主题和同一分区

假设给定主题有100个分区(例如 Purchases )，那么我可以轻松使用这100个分区(例如 Electronics ， Clothing 等).同时使用其中有100个消费者的消费者组.

Suppose I have 100 partitions for a given topic (e.g. Purchases), I can easily consume these 100 partitions (e.g. Electronics, Clothing, and etc...) in parallel using a consumer group with 100 consumers in it.

但是，这会将一个消费者分配给购买上总数据的每个子集.如果我只想与100个使用者同时使用一个数据子集怎么办?例如，对于我所有的消费者，他们只想知道 Purchases 主题的 Electronics 分区.

However, that is assigning one consumer to each subset of the total data on Purchases. What if I want just want to consume one subset of data with 100 consumers concurrently? For example, for all of my consumers, they just want to know Electronics partition of the Purchases topic.

他们可以同时使用此分区吗?

通常，我只希望所有消费者同时接收同一数据集.

In general I just want all my consumers to receive the same data set concurrently.

根据我收集到的信息，在我看来，消费者无法从副本中进行消费:从副本

From the information I've gathered, it seems to me that consumers CANNOT consume from replicas: Consuming from a replica

我可以为多个主题生成相同的数据，例如 Purchase-1 [Electronics] 和 Purchase-2 [Electronics] ，以便随后使用它们并发?这是推荐的方法吗?

Can I produce the same data to multiple topics, like Purchase-1[Electronics] and Purchase-2[Electronics] so then I can consume them concurrently? Is this a recommended approach?

2.同时制作相同主题和相同分区

当多个生产者针对同一个主题和同一个分区进行生产时，由于我们只能写给分区负责人，而副本仅出于容错目的，这是否意味着没有任何并发​​性?

When multiple producers are producing to the same topic and same partition, since we can only write to the partition leader and replicas are only there for fault-tolerance, does this mean there isn't any concurrency? (i.e. each commit must wait in line.)

推荐答案

1. 如果这100个消费者属于不同的消费者组，则他们可以从同一主题进行消费并同时进行分区.在这种情况下，您需要确保每个使用者都能处理100个分区中的负载.
2. 生产者可以同时对同一个主题分区进行生产，但是写入分区的消息的实际顺序由分区负责人确定.

这篇关于卡夫卡的平行生产和消费的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！