使用Kafka Connect时如何转换所有时间戳字段? [英] How to transform all timestamp fields when using Kafka Connect?

问题描述

我正在尝试将所有时间戳字段转换为格式为 yyyy-MM-dd HH:mm:ss 的字符串类型.

要转换多个字段，我必须分别为每个字段创建一个转换.

<预><代码>..."转换":"tsFormat1,tsFormat2,...,tsFormatN","transforms.tsFormat1.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value","transforms.tsFormat1.target.type": "字符串","transforms.tsFormat1.field": "ts_col1","transforms.tsFormat1.format": "yyyy-MM-dd HH:mm:ss","transforms.tsFormat2.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value","transforms.tsFormat2.target.type": "字符串","transforms.tsFormat2.field": "ts_col2","transforms.tsFormat2.format": "yyyy-MM-dd HH:mm:ss",..."transforms.tsFormatN.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value","transforms.tsFormatN.target.type": "字符串","transforms.tsFormatN.field": "ts_colN","transforms.tsFormatN.format": "yyyy-MM-dd HH:mm:ss",...

 

有没有办法对所有时间戳列应用单个转换?

我试过了，

<预><代码>..."转换":"tsFormat","transforms.tsFormat.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value","transforms.tsFormat.target.type": "字符串","transforms.tsFormat.field": "ts_col1, ts_col2,..., ts_colN","transforms.tsFormat.format": "yyyy-MM-dd HH:mm:ss",...

和

<预><代码>..."转换":"tsFormat","transforms.tsFormat.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value","transforms.tsFormat.target.type": "字符串","transforms.tsFormat.field": "ts_col1","transforms.tsFormat.field": "ts_col2",..."transforms.tsFormat.field": "ts_colN","transforms.tsFormat.format": "yyyy-MM-dd HH:mm:ss",...

 

<小时>

更好的是数字类型匹配"numeric.mapping": "best_fit".就像 numeric.mapping 如何应用于所有数字字段(无需手动指定字段名称)以尝试找到最佳数字类型一样，是否有类似的东西可以应用转换或字符串格式对于所有时间戳字段?

解决方案

我在原始 TimestampConverter 的基础上做了一些小调整，如你所愿:https://github.com/howareyouo/kafka-connect-timestamp-converter

I am trying to convert all timestamp fields to a string type with the format yyyy-MM-dd HH:mm:ss.

To transform multiple fields, I have to create a transform for each one individually.

...
"transforms":"tsFormat1,tsFormat2,...,tsFormatN",
"transforms.tsFormat1.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.tsFormat1.target.type": "string",
"transforms.tsFormat1.field": "ts_col1",
"transforms.tsFormat1.format": "yyyy-MM-dd HH:mm:ss",
"transforms.tsFormat2.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.tsFormat2.target.type": "string",
"transforms.tsFormat2.field": "ts_col2",
"transforms.tsFormat2.format": "yyyy-MM-dd HH:mm:ss",
...
"transforms.tsFormatN.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.tsFormatN.target.type": "string",
"transforms.tsFormatN.field": "ts_colN",
"transforms.tsFormatN.format": "yyyy-MM-dd HH:mm:ss",
...

 

Is there any way to apply a single transform on all timestamp columns?

I have tried,

...
"transforms":"tsFormat",
"transforms.tsFormat.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.tsFormat.target.type": "string",
"transforms.tsFormat.field": "ts_col1, ts_col2,..., ts_colN",
"transforms.tsFormat.format": "yyyy-MM-dd HH:mm:ss",
...

and

...
"transforms":"tsFormat",
"transforms.tsFormat.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value",
"transforms.tsFormat.target.type": "string",
"transforms.tsFormat.field": "ts_col1",
"transforms.tsFormat.field": "ts_col2",
...
"transforms.tsFormat.field": "ts_colN",
"transforms.tsFormat.format": "yyyy-MM-dd HH:mm:ss",
...

 

---

What would be even better is something like numeric type matching "numeric.mapping": "best_fit". Just like how numeric.mapping applies to all numeric fields (without having to manually specify the field names) to try and find the best numeric type, is there something like this that can apply a transform or string format for all timestamp fields?

解决方案

I made a little tweaks base on the original TimestampConverter, works as perfect as what you want: https://github.com/howareyouo/kafka-connect-timestamp-converter

这篇关于使用Kafka Connect时如何转换所有时间戳字段?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！