如何在storm中注册kryo序列化器实例? [英] how to register kryo serializer instances in storm?

问题描述

我正在绝望地尝试配置序列化程序实例以在我的 Storm 拓扑中使用.

storm 文档指出，有两种方法可以注册序列化程序:

<块引用>

<代码>1.要注册的类的名称.在这种情况下，Storm 将使用 Kryo 的 FieldsSerializer 来序列化类.这可能是课程的最佳选择，也可能不是 - 有关更多详细信息，请参阅 Kryo 文档.2. 从要注册的类的名称到 com.esotericsoftware.kryo.Serializer 的实现的映射.

我想使用 2.->

<块引用>

MapserializerConfig = new HashMap();serializerConfig.put(Record.class.getName(), new AvroSerializer(params));conf.put(Config.TOPOLOGY_KRYO_REGISTER, serializerConfig);

不幸的是，这导致

<块引用>

线程main"中的异常 java.lang.IllegalArgumentException: Storm conf 无效.必须是 json 可序列化的

关于拓扑提交.

有谁知道怎么做(注册序列化程序实例)?

非常感谢

解决方案

有一个 backtype.storm.Config 类中的方法来注册你自己的从 Serializer 派生的类.

对于您的示例，将其放在创建和提交拓扑的 main 方法中:

//读入 Storm 配置配置 conf = 新配置()；conf.registerSerialization(Record.class, AvroSerializer.class);

I'm desparately trying to configure serializer instances to use in my storm topology.

The storm documentation states, there are 2 ways to register serializers :

1. The name of a class to register. In this case, Storm will use Kryo’s FieldsSerializer to serialize the class. This may or may not be optimal for the class – see the Kryo docs for more details.
2. A map from the name of a class to register to an implementation of com.esotericsoftware.kryo.Serializer.

I want to use 2. ->

Map<String, Object> serializerConfig = new HashMap<String, Object>();
serializerConfig.put(Record.class.getName(), new AvroSerializer(params));
conf.put(Config.TOPOLOGY_KRYO_REGISTER, serializerConfig);

Unfortunately, this results in

Exception in thread "main" java.lang.IllegalArgumentException: Storm conf is not valid. Must be json-serializable

on topology submission.

Does anyone know how to do this (register serializer instances) ?

Thank you very much

解决方案

There is a method in the backtype.storm.Config class to register your own class derived from Serializer.

For your example, put this in the main method that creates and submits the topology:

// Read in Storm Configuration
Config conf = new Config();
conf.registerSerialization(Record.class, AvroSerializer.class);

这篇关于如何在storm中注册kryo序列化器实例?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！