为 kafka 消息实现归档 [英] Implement filering for kafka messages
问题描述
我最近开始使用 Kafka，并针对少数用例评估了 Kafka.
I have started using Kafka recently and evaluating Kafka for few use cases.
如果我们想根据消息内容为消费者(订阅者)提供过滤消息的功能，那么最好的方法是什么?
If we wanted to provide the capability for filtering messages for consumers (subscribers) based on message content, what is best approach for doing this?
假设生产商公开了一个名为交易"的主题，该主题具有不同的交易详细信息，例如市场名称、创建日期、价格等.
Say a topic named "Trades" is exposed by producer which has different trades details such as market name, creation date, price etc.
一些消费者对特定市场的交易感兴趣，而另一些消费者则对特定日期后的交易感兴趣(基于内容的过滤)
Some consumers are interested in trades for a specific markets and others are interested in trades after certain date etc. (content based filtering)
由于代理端无法进行过滤，因此实现以下情况的最佳方法是什么:
As filtering is not possible on broker side, what is best possible approach for implementing below cases :
- 如果过滤条件特定于消费者.我们应该使用消费者拦截器(尽管建议使用拦截器进行日志记录根据文档的目的)?
- 如果过滤标准(基于内容的过滤)在消费者中很常见，那么应该采用什么方法?
在本地监听主题并过滤消息并写入新主题(使用拦截器或流)
Listen to topic and filter the messages locally and write to new topic (using either interceptor or streams)
推荐答案
如果我正确理解您的问题，您有一个主题和不同的消费者，他们对该主题的特定部分感兴趣.同时，您不拥有这些消费者，并希望避免那些消费者只是阅读整个主题并自己进行过滤?
If I understand you question correctly, you have one topic and different consumer which are interested in specific parts of the topic. At the same time, you do not own those consumer and want to avoid that those consumer just read the whole topic and do the filtering by themselves?
为此，构建新应用程序的唯一方法是读取整个主题，进行过滤(或实际拆分)并将数据写回两个(多个)不同的主题.外部消费者将从这些新主题中消费，并且只收到他们感兴趣的日期.
For this, the only way to go it to build a new application, that does read the whole topic, does the filtering (or actually splitting) and write the data back into two (multiple) different topics. The external consumer would consumer from those new topics and only receive the date they are interested in.
为此目的使用 Kafka Streams 将是一个非常好的方法.DSL 应该提供您需要的一切.
Using Kafka Streams for this purpose would be a very good way to go. The DSL should offer everything you need.
作为替代方案，您可以使用 KafkaConsumer 和 KafkaProducer 编写自己的应用程序，在您的用户代码中手动执行过滤/拆分.这与使用 Kafka Streams 没有太大区别，因为 Kafka Streams 应用程序会在内部做完全相同的事情.但是，有了 Streams，您完成它的工作就会少得多.
As an alternative, you can just write your own application using KafkaConsumer and KafkaProducer to do the filtering/splitting manually in your user code. This would not be much different from using Kafka Streams, as a Kafka Streams application would do the exact same thing internally. However, with Streams your effort to get it done would be way less.
我不会为此使用拦截器.即使这行得通，对于您的用例来说，这似乎不是一个好的软件设计.
I would not use interceptors for this. Even is this would work, it seems not to be a good software design for you use case.
这篇关于为 kafka 消息实现归档的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
