从卡夫卡多次读取同一条消息 [英] Reading the same message several times from Kafka

问题描述

我使用 Spring Kafka API 来实现Kafka具有手动偏移管理的消费者:

I use Spring Kafka API to implement Kafka consumer with manual offset management:

@KafkaListener(topics = "some_topic")
public void onMessage(@Payload Message message, Acknowledgment acknowledgment) {
    if (someCondition) {
        acknowledgment.acknowledge();
    }
}

在这里，我希望使用者仅在someCondition成立的情况下才提交偏移量.否则，消费者应该睡一段时间，然后再次阅读同一则消息.

Here, I want the consumer to commit the offset only if someCondition holds. Otherwise the consumer should sleep for some time and read the same message again.

Kafka配置:

@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(new DefaultKafkaConsumerFactory<>(consumerConfig());
    factory.getContainerProperties().setAckMode(MANUAL);
    return factory;
}

private Map<String, Object> consumerConfig() {
    Map<String, Object> props = new HashMap<>();
    ...
    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
    ...
    return props;
}

在当前配置下，如果为someCondition == false，则使用者不会提交偏移量，但仍会读取下一条消息.如果未执行Kafka acknowledgement，是否可以使消费者重新阅读消息?

With the current configuration, if someCondition == false, consumer doesn't commit the offset, but still reads the next messages. Is there a way to make the consumer reread a message if the Kafka acknowledgement wasn't performed?

推荐答案

您可以停止并重新启动容器，然后将其重新发送.

You can stop and restart the container and it will be re-sent.

在即将发布的1.1版本中，您可以搜索到所需的偏移量，它将重新发送.

With the upcoming 1.1 release, you can seek to the required offset and it will be resent.

但是，如果以后的消息已经被检索，您仍然会首先看到它们，因此您也必须丢弃这些消息.

But you will still see later messages first if they have already been retrieved so you will have to discard those too.

第二个里程碑具有该功能，我们预计将在下周发布.

The second milestone has that feature and we expect it to be released next week.

这篇关于从卡夫卡多次读取同一条消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！