从 Kafka 请求两个时间戳之间的消息 [英] Request messages between two timestamps from Kafka

问题描述

是否可以根据消息被摄取的时间段来消费来自 Kafka 的消息?

Is it possible to consume messages from Kafka based on a time period in which the messages were ingested?

示例:我希望将所有消息提取到今天 0900-1000 之间的主题(现在是 1200).

Example: I want all messages ingested to a topic between 0900-1000 today (and now it's 1200).

如果只有一种方法可以指定开始时间，那很好 - 我的消费者可以在到达结束时间后停止处理消息.

If there is only a way to specify a start time, that's fine - my consumer can stop processing messages once it reaches the end time.

我可以看到从给定偏移量请求消息的方法，以及获取第一个可用偏移量和最早可用偏移量的方法，但不是给定时间后的所有消息.

I can see methods for requesting messages from a given offset, and for getting the first available offset, and for the earliest available offset, but not all messages after a given time.

推荐答案

您可以使用 offsetsForTimes 方法返回时间戳大于或等于给定时间戳的偏移量.有关官方文档的更多信息:

You could use the offsetsForTimes method which returns you offset whose timestamp is greater or equal to the give timestamp. More information on the official doc here :

https://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#offsetsForTimes(java.util.Map)

获得偏移量后，您可以寻求使用它并从那里开始读取.

After getting the offset you can seek using it and starting to read from there.

这篇关于从 Kafka 请求两个时间戳之间的消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！