融合的s3接收器将密钥放在哪里？ [英] where does confluent s3 sink put the key?

问题描述

我设置了一个汇合的s3接收器连接，它将.avro文件存储在s3中。

我转储了这些文件，发现它们只是消息本身，我不知道该在哪里找到消息密钥，知道吗？

I setup a confluent s3 sink connect, it stores .avro files in s3.
I dump those files, and find out that they are just the message itself, I don't know where can I find the message key, any idea?

配置类似于：

{
    "name": "s3-sink-test",
    "config": {
        "connector.class": "io.confluent.connect.s3.S3SinkConnector",
        "tasks.max": "1",
        "topics": "book",
        "s3.region": "eu-central-1",
        "s3.bucket.name": "kafka",
        "s3.part.size": "5242880",
        "storage.class": "io.confluent.connect.s3.storage.S3Storage",
        "format.class": "io.confluent.connect.s3.format.avro.AvroFormat",
        "schema.generator.class": "io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator",
        "partitioner.class": "io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
        "path.format": "'year'=YYYY/'month'=MM/'day'=dd/'hour'=HH",
        "locale": "US",
        "timezone": "UTC",
        "partition.duration.ms": "3600000",
        "timestamp.extractor": "RecordField",
        "timestamp.field": "local_timestamp",
        "flush.size": "2",
        "schema.compatibility": "NONE"
    }
}

推荐答案

开箱即用的任何存储设备都不能使用Kafka消息密钥

The Kafka message key is not persevered by any of the storage Kafka Connectors by Confluent out of the box

尝试编译并设置存档转换，可以使用连接配置中的这些属性进行设置

Try compiling and setting up the Archive Transform, which can be setup using these properties in the Connect Configuration

"transforms" : "Archive",
"transforms.Archive.type" : "com.github.jcustenborder.kafka.connect.archive.Archive"

有关Kafka Connect中SMT的更多信息，请参见此博客文章

For more about SMTs in Kafka Connect, see this blog post

这篇关于融合的s3接收器将密钥放在哪里？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！