我们使用使用者组通过多个使用者从经纪人读取数据，但是如何合并使用的数据? [英] We read data from brokers through multiple consumers using consumer group, but how the consumed data is combined?

问题描述

我需要来自kafka经纪人的数据，但是为了快速访问，我正在使用多个具有相同组ID的消费者(称为消费者组).但是在每个消费者阅读之后，我们如何合并来自多个消费者的数据?有逻辑吗?

I need data from kafka brokers,but for fast access I am using multiple consumers with same group id known as consumer groups.But after reading by each consumer,how can we combine data from multiple consumers? Is there any logic?

推荐答案

通过设计，同一使用者组中的不同使用者彼此独立地处理数据. (这种行为使应用程序可以很好地扩展.)

By design, different consumers in the same consumer group process data independently from each other. (This behavior is what allows applications to scale well.)

但是在每个消费者阅读之后，我们如何合并来自多个消费者的数据?有逻辑吗?

But after reading by each consumer,how can we combine data from multiple consumers? Is there any logic?

使用Kafka的"Consumer API"(也称为"consumer client"库)时，答案很简短，但略有简化.根据问题的措辞，我认为这是您正在使用的方式:如果需要合并数据对于来自多个使用者的最简单的选择，是在另一个Kafka主题中提供此(新)输入数据，在该主题中，您将在后续处理步骤中进行合并.一个简单的例子是:将另一个第二个Kafka主题设置为只有1个分区，因此任何后续处理步骤都将看到所有需要合并的数据.

The short but slightly simplified answer when you use Kafka's "Consumer API" (also called: "consumer client" library), which I think is what you are using based on the wording of your question: If you need to combine data from multiple consumers, the easiest option is to make this (new) input data available in another Kafka topic, where you do the combining in a subsequent processing step. A trivial example would be: the other, second Kafka topic would be set up to have just 1 partition, so any subsequent processing step would see all the data that needs to be combined.

如果听起来有点太复杂了，我建议使用Kafka的Streams API，这样可以更轻松地定义此类处理流程(例如，联接或聚合，如您所提的问题).换句话说，Kafka Streams为您提供了许多您想要的内置逻辑":

If this sounds a bit too complicated, I'd suggest to use Kafka's Streams API, which makes it much easier to define such processing flows (e.g. joins or aggregations, like in your question). In other words, Kafka Streams gives you a lot of the desired built-in "logic" that you are looking for: https://kafka.apache.org/documentation/streams/

这篇关于我们使用使用者组通过多个使用者从经纪人读取数据，但是如何合并使用的数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！