为什么 Shopify Sarama 消费者需要分区来消费消息 [英] Why partition is needed in Shopify Sarama consumer to consume messages

问题描述

我很抱歉发布与 Kafka 图书馆相关的问题，因为没有多少人对图书馆的特定问题感兴趣.但是这个库是 golang-Kafka 实现中最常用的库之一.

I am sorry for posting a question related to a Kafka Library as not many people are interested in Library specific questions. But this library is one of the most used library for golang-Kafka implementations.

我想使用 Sarama 库创建一个简单的消费者，它可以监听一个主题.现在据我所知，在高级 Kafka API 中，如果未指定特定分区，默认情况下消费者会侦听所有主题分区.但是，在这个库中，Consumer 接口只有需要分区参数的 ConsumePartition 函数.函数签名为:

I want to create a simple consumer using Sarama library which listens to a topic. Now as far as I know, in the high-level Kafka API's, by default a consumer listens to all the topics partitions if a specific partition is not specified. However, in this Library, the Consumer interface has only ConsumePartition function where the partition is required param. The signature of function is:

ConsumePartition(topic string, partition int32, offset int64) (PartitionConsumer, error) 

这让我有点困惑.有做过这方面工作的人吗?

This confuses me a bit. Anyone who has worked on it?

另外，我有一个关于 Kafka 的基本问题.如果我有一个由 3 个消费者实例组成的消费者组，并且他们正在听让我们说 2 个主题，每个主题有 2 个分区，那么我是否需要特别提及哪个消费者实例将消费到哪个分区或 Kafka Fetch API 将处理它它自己的基于负载?

Also, I have a basic question regarding Kafka. If I have a consumer group consisting of 3 consumer instances and they are listening to let's say 2 topics each having 2 partitions, then do I need to specifically mention which consumer instance will consume to which partition or Kafka Fetch API will take care of it on its own based on load?

推荐答案

我使用 sarama-cluster 这是 Sarama 的开源扩展(也被 Shopify Sarama 推荐)).
借助 Sarama 集群，您可以使用此 API 创建消费者:

I use sarama-cluster which is an open source extension for Sarama (also recommended by Shopify Sarama).
With Sarama cluster you can create a consumer using this API:

cluster.NewConsumer(brokers, consumerGroup, topics, kafkaConfig)

所以不需要分区.你应该只提供你的 Kafka brokers 的地址，你的 consumer group 的名称以及你想要消费的 topics.

so no partition is needed. You should only provide the addresses of your Kafka brokers, the name of your consumer group and which topics you wish to consume.

消费者处理

为了维护秩序，你应该只为每个分区分配一个消费者.
因此，如果您的消费者组中有 3 个消费者，并且您希望他们消费 2 个主题，每个主题有 2 个分区，您应该按如下方式分配:

To maintain order you should assign to each partition only one consumer.
So in case you have 3 consumers in your consumer group and you want them to consume 2 topics having 2 partitions each, you should assign as follows:

partitions 1,2 -> consumer A  
partition 3 -> consumer B  
partition 4 -> consumer C 

您最终可能会发现其中一个消费者进展更快(其中一个主题的吞吐量更高)，您需要重新平衡.
建议使用为您处理此问题的库(如 sarama-cluster).

You might end up with one of the consumers advancing faster (one of the topics have higher throughput) and you will need to re-balance.
Using a library (like sarama-cluster) that handles this for you is recommended.

这篇关于为什么 Shopify Sarama 消费者需要分区来消费消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！