消费者.如何指定要读取的分区?[卡夫卡] [英] consumer.How to specify partition to read? [kafka]

问题描述

我是用 kafka 介绍的，我想知道当我使用来自主题的消息时如何指定分区.

I am introducing with kafka and I want to know how to specify partition when I consume messages from topic.

我找到了几张这样的图片:

I have found several picture like this:

表示1个consumer可以消费多个partition的消息，但是1个partition可以被单个consumer(在consumer group内)读取

It means that 1 consumer can consume messages from several partitions but 1 partition can be read by single consumer(within consumer group)

另外，我已经阅读了几个消费者示例，它看起来像这样:

Also I have read several examples for consumer and it looks like this:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "consumer-tutorial");
props.put("key.deserializer", StringDeserializer.class.getName());
props.put("value.deserializer", StringDeserializer.class.getName());
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props); 

和:

consumer.subscribe(Arrays.asList("foo", "bar")); 

2.投票

 try {
      while (running) {
        ConsumerRecords<String, String> records = consumer.poll(1000);
        for (ConsumerRecord<String, String> record : records)
          System.out.println(record.offset() + ": " + record.value());
      }
    } finally {
      consumer.close();
    }

这是如何工作的?我将从哪个分区读取消息?

How does this work? From which partition will I read messages?

推荐答案

有两种方法可以告诉您要使用哪个主题/分区:KafkaConsumer#assign()(你指定你想要的分区和偏移量您开始)和订阅(您加入一个消费者组，分区/偏移量将由组协调器根据同一消费者组中的消费者动态分配，并可能在运行时更改)

There are two ways to tell what topic/partitions you want to consume: KafkaConsumer#assign() (you specify the partition you want and the offset where you begin) and subscribe (you join a consumer group, and partition/offset will be dynamically assigned by group coordinator depending of consumers in the same consumer group, and may change during runtime)

在这两种情况下，您都需要poll来接收数据.

In both case, you need to poll to receive data.

参见 https:///kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html，尤其是段落Consumer Groups and Topic Subscriptions和手动分区分配

这篇关于消费者.如何指定要读取的分区?[卡夫卡]的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！