查询 Kafka 主题以获取特定记录 [英] Query Kafka topic for specific record

问题描述

是否有一种优雅的方式来查询特定记录的 Kafka 主题?我正在构建的 REST API 获取一个 ID，并需要在 Kafka 主题中查找与该 ID 关联的记录.一种方法是通过自定义使用者检查主题中的每条记录并查找匹配项，但我想避免读取一堆记录的开销.Kafka 是否具有快速的内置过滤功能?

Is there an elegant way to query a Kafka topic for a specific record? The REST API that I'm building gets an ID and needs to look up records associated with that ID in a Kafka topic. One approach is to check every record in the topic via a custom consumer and look for a match, but I'd like to avoid the overhead of reading a bunch of records. Does Kafka have a fast, built in filtering capability?

推荐答案

在 Kafka 中搜索记录的唯一快速方法(过度简化)是通过分区和偏移.新的生产者类可以通过期货返回写入消息的分区和偏移量.您可以使用这两个值来非常快速地检索消息.

The only fast way to search for a record in Kafka (to oversimplify) is by partition and offset. The new producer class can return, via futures, the partition and offset into which a message was written. You can use these two values to very quickly retrieve the message.

因此，如果您将 ID 从分区和偏移量中提取出来，那么您就可以实现快速查询.否则，没有那么多.这意味着对象的 ID 不是您的数据模型的一部分，而是由 Kafka-knowledgable 代码生成的.

So if you make the ID out of the partition and offset then you can implement your fast query. Otherwise, not so much. This means that the ID for an object isn't part of your data model, but rather is generated by the Kafka-knowledgable code.

也许这对你有用，也许不.

Maybe that works for you, maybe it doesn't.

这篇关于查询 Kafka 主题以获取特定记录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！