为什么Shopify Sarama消费者需要分区才能使用消息 [英] Why partition is needed in Shopify Sarama consumer to consume messages

问题描述

很抱歉，我发布了一个与Kafka图书馆有关的问题，因为对图书馆特定问题不感兴趣的人并不多.但是，该库是用于golang-Kafka实现的最常用的库之一.

I am sorry for posting a question related to a Kafka Library as not many people are interested in Library specific questions. But this library is one of the most used library for golang-Kafka implementations.

我想使用Sarama库创建一个简单的使用者，该库可以侦听某个主题.现在，据我所知，在高级Kafka API中，默认情况下，如果未指定特定分区，则使用者将收听所有主题分区.但是，在此库中，消费者接口仅具有ConsumePartition函数，在该函数中分区是必需参数.函数的签名是:

I want to create a simple consumer using Sarama library which listens to a topic. Now as far as I know, in the high-level Kafka API's, by default a consumer listens to all the topics partitions if a specific partition is not specified. However, in this Library, the Consumer interface has only ConsumePartition function where the partition is required param. The signature of function is:

ConsumePartition(topic string, partition int32, offset int64) (PartitionConsumer, error) 

这让我有些困惑.有人做过吗?

This confuses me a bit. Anyone who has worked on it?

此外，我还有一个关于Kafka的基本问题.如果我有一个由3个使用者实例组成的使用者组，并且他们正在听每个包含2个分区的2个主题，那么我是否需要特别提及哪个使用者实例将消耗到哪个分区，或者Kafka Fetch API将在该实例上进行处理自己基于负载?

Also, I have a basic question regarding Kafka. If I have a consumer group consisting of 3 consumer instances and they are listening to let's say 2 topics each having 2 partitions, then do I need to specifically mention which consumer instance will consume to which partition or Kafka Fetch API will take care of it on its own based on load?

推荐答案

我使用 sarama-cluster 是Sarama的开源扩展(也被 Shopify Sarama 推荐)).
借助Sarama集群，您可以使用以下API创建使用者:

I use sarama-cluster which is an open source extension for Sarama (also recommended by Shopify Sarama).
With Sarama cluster you can create a consumer using this API:

cluster.NewConsumer(brokers, consumerGroup, topics, kafkaConfig)

因此不需要分区.您只应提供Kafka经纪人的地址，经纪人组的名称以及您想消费的主题.

so no partition is needed. You should only provide the addresses of your Kafka brokers, the name of your consumer group and which topics you wish to consume.

消费者处理

要维持顺序，您应该仅将一个使用者分配给每个分区.
因此，如果您的消费者组中有3个消费者，并且您希望他们消费2个主题，每个主题有2个分区，则应按以下方式分配:

To maintain order you should assign to each partition only one consumer.
So in case you have 3 consumers in your consumer group and you want them to consume 2 topics having 2 partitions each, you should assign as follows:

partitions 1,2 -> consumer A  
partition 3 -> consumer B  
partition 4 -> consumer C 

您最终可能会导致其中一位消费者前进得更快(其中一个主题具有更高的吞吐量)，并且您将需要重新平衡.
建议使用可以为您处理此问题的库(例如sarama-cluster).

You might end up with one of the consumers advancing faster (one of the topics have higher throughput) and you will need to re-balance.
Using a library (like sarama-cluster) that handles this for you is recommended.

这篇关于为什么Shopify Sarama消费者需要分区才能使用消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！