在Avro文件中存储空值 [英] Storing null values in avro files
问题描述
我有一些看起来像这样的json数据:
I have some json data that looks like this:
{
"id": 1998983092,
"name": "Test Name 1",
"type": "search string",
"creationDate": "2017-06-06T13:49:15.091+0000",
"lastModificationDate": "2017-06-28T14:53:19.698+0000",
"lastModifiedUsername": "testuser@test.com",
"lockedQuery": false,
"lockedByUsername": null
}
我能够毫无问题地将lockedQuery null值添加到GenericRecord对象.
I am able to add the lockedQuery null value to a GenericRecord object without issue.
GenericRecord record = new GenericData.Record(schema);
if(json.isNull("lockedQuery")){
record.put("lockedQuery", null);
}
但是,稍后当我尝试将GenericRecord对象写入avro文件时,会出现空指针异常.
However, later when I attempt to write that GenericRecord object to an avro file I get a null pointer exception.
File file = new File("~/test.arvo");
DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<>(schema);
DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<>(datumWriter);
dataFileWriter.create(schema, file);
for(GenericRecord record: masterList) {
dataFileWriter.append(record); // NULL POINTER HERE
}
运行该代码时,出现以下异常.非常感谢有关如何将空值处理到Avro文件中的任何提示.预先感谢.
When I run that code I get the following exception. Any tips on how to process a null value into an Avro file much appreciated. Thanks in advance.
java.lang.NullPointerException: null of boolean in field lockedQuery of
com.mydomain.test1.domain.MyAvroRecord
Exception in thread "main" java.lang.RuntimeException:
org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.NullPointerException: null of boolean in field lockedQuery of
com.mydomain.test1.domain.MyAvroRecord
at com.mydomain.avro.App.main(App.java:198)
Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.NullPointerException: null of boolean in field lockedQuery of
com.mydomain.test1.domain.MyAvroRecord
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
这是MyAvroRecord
here is the MyAvroRecord
public class MyAvroRecord {
long id;
String name;
String type;
Date timestamp;
Date lastModifcationDate;
String lastModifiedUsername;
Boolean lockedQuery;
推荐答案
要能够将Avro字段设置为null
,应在Avro模式中允许此操作,方法是将null
添加为字段的可能类型之一.看一下Avro文档中的示例:
To be able to set Avro field to null
you should allow this in Avro schema, by adding null
as one of the possible types of the field. Take a look on example from Avro documentation:
{
"type": "record",
"name": "MyRecord",
"fields" : [
{"name": "userId", "type": "long"}, // mandatory field
{"name": "userName", "type": ["null", "string"]} // optional field
]
}
此处userName
被声明为复合类型,可以是null
或string
.这种定义允许将userName
字段设置为null.由于对比度userId
只能包含长值,因此尝试将userId
设置为null会导致NullPointerException
.
here userName
is declared as composite type which could be either null
or string
. This kind of definition allows to set userName
field to null. As contrast userId
can only contain long values, hence attempt to set userId
to null will result in NullPointerException
.
这篇关于在Avro文件中存储空值的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!