Kafka Streams:POJO 序列化/反序列化 [英] Kafka Streams: POJO serialization/deserialization

问题描述

我们可以使用 Kafka Streams 中的什么类/方法将 Java 对象序列化/反序列化为字节数组，反之亦然?以下链接提出了ByteArrayOutputStream & 的用法.ObjectOutputStream 但它们不是线程安全的.

What class/method in Kafka Streams can we use to serialize/deserialize Java object to byte array OR vice versa? The following link proposes the usage of ByteArrayOutputStream & ObjectOutputStream but they are not thread safe.

将自定义 Java 对象发送到 Kafka 主题

还有另一个选项可以使用 ObjectMapper，ObjectReader(用于线程安全)，但这是从 POJO -> JSON -> bytearray 转换而来的.似乎这个选项是一个广泛的选项.想检查是否有直接的方法将对象转换为字节数组，反之亦然，这是线程安全的.请推荐

There is another option to use the ObjectMapper, ObjectReader (for threadsafe), but that's converting from POJO -> JSON -> bytearray. Seems this option is an extensive one. Wanted to check if there is a direct way to translate object into bytearray and vice versa which is threadsafe. Please suggest

import org.apache.kafka.common.serialization.Serializer;
public class HouseSerializer<T> implements Serializer<T>{
    private Class<T> tClass;
    public HouseSerializer(){

    }

    @SuppressWarnings("unchecked")
    @Override
    public void configure(Map configs, boolean isKey) {
        tClass = (Class<T>) configs.get("POJOClass");       
    }

    @Override
    public void close() {
    }

    @Override
    public byte[] serialize(String topic, T data) {
        //Object serialization to be performed here
        return null;
    }
}

注意:Kafka 版本 - 0.10.1

Note: Kafka version - 0.10.1

推荐答案

想检查是否有将对象转换为字节数组的直接方法

Wanted to check if there is a direct way to translate object into bytearray

我建议您将 Avro 序列化 与 Confluent Schema Registry 一起使用，如果可能，但不是必需的.JSON 是一个很好的回退，但在线"占用更多空间，因此 MsgPack 将是那里的替代方案.

I would suggest you look at using Avro serialization with the Confluent Schema Registry, if possible, but not required. JSON is a good fall back, but takes more space "on the wire", and so MsgPack would be the alternative there.

在此处查看 Avro 代码示例

以上示例使用 avro-maven-plugin从 src/main/resources/avro 模式文件生成一个 LogLine 类.

Above example is using the avro-maven-plugin to generate a LogLine class from the src/main/resources/avro schema file.

否则，由您决定 如何将你的对象序列化为字节数组，例如一个String通常被打包为

Otherwise, it's up to you for how to serialize your object into a byte array, for example, a String is commonly packed as

[(length of string) (UTF8 encoded bytes)]

虽然布尔值是单个 0 或 1 位

While booleans are a single 0 or 1 bit

线程安全

我理解这种担忧，但您通常不会在线程之间共享反序列化数据.您为每个独立的消息发送/读取/处理消息.

I understand the concern, but you aren't commonly sharing deserialized data between threads. You send/read/process a message for each independent one.

这篇关于Kafka Streams:POJO 序列化/反序列化的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！