confluent s3 sink 把钥匙放在哪里? [英] where does confluent s3 sink put the key?
问题描述
我设置了一个融合的 s3 接收器连接,它将 .avro 文件存储在 s3 中.
我转储这些文件,发现它们只是消息本身,我不知道在哪里可以找到消息密钥,知道吗?
I setup a confluent s3 sink connect, it stores .avro files in s3.
I dump those files, and find out that they are just the message itself, I don't know where can I find the message key, any idea?
配置如下:
{
"name": "s3-sink-test",
"config": {
"connector.class": "io.confluent.connect.s3.S3SinkConnector",
"tasks.max": "1",
"topics": "book",
"s3.region": "eu-central-1",
"s3.bucket.name": "kafka",
"s3.part.size": "5242880",
"storage.class": "io.confluent.connect.s3.storage.S3Storage",
"format.class": "io.confluent.connect.s3.format.avro.AvroFormat",
"schema.generator.class": "io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator",
"partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
"path.format": "'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH",
"locale": "US",
"timezone": "UTC",
"partition.duration.ms": "3600000",
"timestamp.extractor": "RecordField",
"timestamp.field": "local_timestamp",
"flush.size": "2",
"schema.compatibility": "NONE"
}
}
推荐答案
Edit 代码已更改以启用此功能,并且不需要以下转换 - docs
Edit The code has changed to enable this, and the below transform isn't needed - docs
开箱即用的 Confluent 的任何存储 Kafka 连接器都不会保留 Kafka 消息密钥
尝试编译设置Archive Transform,可以设置在连接配置中使用这些属性
Try compiling and setting up the Archive Transform, which can be setup using these properties in the Connect Configuration
"transforms" : "Archive",
"transforms.Archive.type" : "com.github.jcustenborder.kafka.connect.archive.Archive"
有关 Kafka Connect 中 SMT 的更多信息,请参阅 这篇博文
For more about SMTs in Kafka Connect, see this blog post
这篇关于confluent s3 sink 把钥匙放在哪里?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!