Kafka Connect-在写入接收器之前修改记录 [英] Kafka Connect- Modifying records before writing into sink

问题描述

我已经使用 confluent-4.0.0 安装了 Kafka 连接使用 hdfs 连接器，我可以将从 Kafka 主题收到的 Avro 记录保存到配置单元.我想知道在写入 hdfs 接收器之前是否有任何方法可以修改记录.我的要求是对记录的值进行小的修改.例如，对整数执行算术运算或操作字符串等.请建议是否有任何方法可以实现这一目标

I have installed Kafka connect using confluent-4.0.0 Using hdfs connector I am able to save Avro records received from Kafka topic to hive. I would like to know if there is any way to modify the records before writing into hdfs sink. My requirement is to do small modifications to values of the record. For Example, performing arithmetic operations on integers or manipulation of strings etc. Please suggest if there any way to achieve this

推荐答案

您有多种选择.

1. 单个消息转换，您可以在操作中看到此处.当消息通过 Connect 时，非常适合轻量级更改.基于配置文件，可使用 提供的扩展API 如果没有现有的转换可以执行您想要的操作.

1. Single Message Transforms, which you can see in action here. Great for light-weight changes as messages pass through Connect. Configuration-file based, and extensible using the provided API if there's not an existing transform that does what you want.

请参阅此处的讨论SMT 何时适合给定要求.

See the discussion here on when SMT are suitable for a given requirement.

KSQL 是 Kafka 的流式 SQL 引擎.在将数据流发送到 HDFS 之前，您可以使用它来修改数据流.在此处查看此示例.

KSQL is a streaming SQL engine for Kafka. You can use it to modify your streams of data before sending them to HDFS. See this example here.

KSQL 建立在 Kafka Stream 的 API，它是一个 Java 库，可让您随心所欲地转换数据.这是一个例子.

KSQL is built on the Kafka Stream's API, which is a Java library and gives you the power to transform your data as much as you'd like. Here's an example.

这篇关于Kafka Connect-在写入接收器之前修改记录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！