从 Kafka 多次读取相同的消息 [英] Reading the same message several times from Kafka

问题描述

我使用Spring Kafka API来实现Kafka具有手动偏移管理的消费者:

I use Spring Kafka API to implement Kafka consumer with manual offset management:

@KafkaListener(topics = "some_topic")
public void onMessage(@Payload Message message, Acknowledgment acknowledgment) {
    if (someCondition) {
        acknowledgment.acknowledge();
    }
}

在这里，我希望消费者仅在 someCondition 成立时提交偏移量.否则消费者应该睡一段时间并再次阅读相同的消息.

Here, I want the consumer to commit the offset only if someCondition holds. Otherwise the consumer should sleep for some time and read the same message again.

Kafka 配置:

@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
    ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(new DefaultKafkaConsumerFactory<>(consumerConfig());
    factory.getContainerProperties().setAckMode(MANUAL);
    return factory;
}

private Map<String, Object> consumerConfig() {
    Map<String, Object> props = new HashMap<>();
    ...
    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
    ...
    return props;
}

以当前配置，如果someCondition == false，消费者不提交偏移量，但仍读取下一条消息.如果 Kafka acknowledgement 没有执行，有没有办法让消费者重新阅读消息?

With the current configuration, if someCondition == false, consumer doesn't commit the offset, but still reads the next messages. Is there a way to make the consumer reread a message if the Kafka acknowledgement wasn't performed?

推荐答案

你可以停止并重启容器，它会被重新发送.

You can stop and restart the container and it will be re-sent.

随着即将发布的 1.1 版本，您可以寻找所需的偏移量，它将被重新发送.

With the upcoming 1.1 release, you can seek to the required offset and it will be resent.

但是，如果已经检索到了后面的消息，您仍然会首先看到它们，因此您也必须丢弃这些消息.

But you will still see later messages first if they have already been retrieved so you will have to discard those too.

第二个里程碑具有该功能，我们预计将于下周发布.

The second milestone has that feature and we expect it to be released next week.

这篇关于从 Kafka 多次读取相同的消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！