KafkaConsumer Java API subscribe() 与assign() [英] KafkaConsumer Java API subscribe() vs assign()

问题描述

我是 Kafka Java API 的新手，我正在处理来自特定 Kafka 主题的记录.

I am new with Kafka Java API and I am working on consuming records from a particular Kafka topic.

我知道我可以使用方法 subscribe() 从主题开始轮询记录.如果我想从主题的选定分区开始轮询记录，Kafka 还提供了方法 assign().

I understand that I can use method subscribe() to start polling records from the topic. Kafka also provides method assign() if I want to start polling records from selected partitions of the topics.

我想了解这是否是两者之间的唯一区别?

I want to understand if this is the only difference between the two?

推荐答案

是 subscribe 需要 group.id 因为组中的每个消费者都会动态分配到分区列表subscribe 方法中提供的主题数量，并且每个分区都可以由该组中的一个消费者线程使用.这是通过平衡消费者组中所有成员之间的分区来实现的，以便将每个分区分配给组中的一个消费者

Yes subscribe need group.id because each consumer in a group will dynamically assigned to partitions for list of topics provided in subscribe method and each partition can be consumed by one consumer thread in that group. This is achieved by balancing the partitions between all members in the consumer group so that each partition is assigned to exactly one consumer in the group

assign 将手动分配一个分区列表给这个消费者.并且此方法不使用消费者的组管理功能(不需要group.id)

assign will manually assign a list of partitions to this consumer. and this method does not use the consumer's group management functionality (where no need of group.id)

主要区别在于 assign(Collection) 会在动态分区分配和消费者组协调上松散控制器

The main difference is assign(Collection) will loose the controller over dynamic partition assignment and consumer group coordination

消费者也可以使用assign(Collection)手动分配特定分区(类似于旧的简单"消费者).在这种情况下，动态分区分配和消费者组协调将被禁用.

It is also possible for the consumer to manually assign specific partitions (similar to the older "simple" consumer) using assign(Collection). In this case, dynamic partition assignment and consumer group coordination will be disabled.

订阅

public void subscribe(java.util.Collection<java.lang.String> topics)

订阅方法订阅给定的主题列表以获取动态分配的分区.如果给定的主题列表为空，则将其视为与 unsubscribe() 相同.

The subscribe method Subscribe to the given list of topics to get dynamically assigned partitions. and if the given list of topics is empty, it is treated the same as unsubscribe().

作为组管理的一部分，消费者将跟踪属于特定组的消费者列表，并在以下事件之一触发时触发重新平衡操作 -

As part of group management, the consumer will keep track of the list of consumers that belong to a particular group and will trigger a rebalance operation if one of the following events trigger -

Number of partitions change for any of the subscribed list of topics
Topic is created or deleted
An existing member of the consumer group dies
A new member is added to an existing consumer group via the join API

分配

public void assign(java.util.Collection<TopicPartition> partitions)

assign 方法手动为这个消费者分配一个分区列表.如果给定的主题分区列表为空，则将其视为与 unsubscribe() 相同.

The assign method manually assign a list of partitions to this consumer. And if the given list of topic partitions is empty, it is treated the same as unsubscribe().

通过此方法手动分配主题不使用消费者的组管理功能.因此，当组成员或集群和主题元数据发生变化时，不会触发重新平衡操作.

Manual topic assignment through this method does not use the consumer's group management functionality. As such, there will be no rebalance operation triggered when group membership or cluster and topic metadata change.

这篇关于KafkaConsumer Java API subscribe() 与assign()的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！