Kafka connect(单消息变换)行过滤 [英] Kafka connect (Single message transform) row filtering
问题描述
我阅读了 kafka 0.10.2.1 中引入的 Kafka 连接转换https://kafka.apache.org/documentation/#connect_transforms
I read about Kafka connect transformations introduced in kafka 0.10.2.1 https://kafka.apache.org/documentation/#connect_transforms
我注意到所有转换都是基于列的转换.我有一个用例,我需要基于值的过滤.例如:
I noticed that all the transformations are column based transformations. I have a use-case where I need value based filtering. For example:
考虑以下一组人的数据集:
consider the following dataset of a group of people:
{"firstName": "FirstName1", "lastName": "LastName1", "age": 30}
{"firstName": "FirstName2", "lastName": "LastName2", "age": 30}
{"firstName": "FirstName3", "lastName": "LastName1", "age": 60}
{"firstName": "FirstName4", "lastName": "LastName2", "age": 60}
我希望我的工作人员过滤所有姓氏为 LastName2 的记录
I want my worker to filter all those records whose lastName is LastName2
是否可以使用 kafka-connect 或者我需要为此用例编写单独的程序.
Is it possible using kafka-connect or I need to write a separate program for this use-case.
谢谢
推荐答案
没有理由不能使用 Single Message Transforms 解决此问题 - 但您需要编写自定义转换,因为您所描述的内容不可用通过当前交付的转换.
No reason why you couldn't solve this with Single Message Transforms - but you'd need to write a custom one since what you're describing is not available through the transforms currently shipped.
这是关于何时使用和不使用 SMT 的有用讨论:2017 年纽约 Kafka 峰会:单消息转换不是您正在寻找的转换(Ewen Cheslack-Postava,Confluent)
This is a useful talk here on when to use, and not to use, SMTs: Kafka Summit New York 2017 : Single Message Transformations Are Not the Transformations You’re Looking For (Ewen Cheslack-Postava, Confluent)
2020 年 4 月编辑:现在有一个 过滤 SMT 作为 Confluent Platform 的一部分提供
Edit April 2020: There is now a Filter SMT provided as part of Confluent Platform
这篇关于Kafka connect(单消息变换)行过滤的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!