在Avro文件中存储空值 [英] Storing null values in avro files

问题描述

我有一些看起来像这样的json数据:

I have some json data that looks like this:

  {
    "id": 1998983092,
    "name": "Test Name 1",
    "type": "search string",
    "creationDate": "2017-06-06T13:49:15.091+0000",
    "lastModificationDate": "2017-06-28T14:53:19.698+0000",
    "lastModifiedUsername": "[e-mail]",
    "lockedQuery": false,
    "lockedByUsername": null
  }

我能够毫无问题地将lockedQuery null值添加到GenericRecord对象.

I am able to add the lockedQuery null value to a GenericRecord object without issue.

GenericRecord record = new GenericData.Record(schema);
if(json.isNull("lockedQuery")){
    record.put("lockedQuery", null);
} 

但是，稍后当我尝试将GenericRecord对象写入avro文件时，会出现空指针异常.

However, later when I attempt to write that GenericRecord object to an avro file I get a null pointer exception.

File file = new File("~/test.arvo");
DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<>(schema);
DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<>(datumWriter);
dataFileWriter.create(schema, file);
for(GenericRecord record: masterList) {
    dataFileWriter.append(record); // NULL POINTER HERE
}

运行该代码时，出现以下异常.非常感谢有关如何将空值处理到Avro文件中的任何提示.预先感谢.

When I run that code I get the following exception. Any tips on how to process a null value into an Avro file much appreciated. Thanks in advance.

java.lang.NullPointerException: null of boolean in field lockedQuery of 
com.mydomain.test1.domain.MyAvroRecord
Exception in thread "main" java.lang.RuntimeException: 
org.apache.avro.file.DataFileWriter$AppendWriteException: 
java.lang.NullPointerException: null of boolean in field lockedQuery of 
com.mydomain.test1.domain.MyAvroRecord
at com.mydomain.avro.App.main(App.java:198)
Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: 
java.lang.NullPointerException: null of boolean in field lockedQuery of 
com.mydomain.test1.domain.MyAvroRecord
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)

这是MyAvroRecord

here is the MyAvroRecord

public class MyAvroRecord {
    long id;
    String name;
    String type;
    Date timestamp;
    Date lastModifcationDate;
    String lastModifiedUsername;
    Boolean lockedQuery;

推荐答案

要能够将Avro字段设置为null，应在Avro模式中允许此操作，方法是将null添加为字段的可能类型之一.看一下Avro文档中的示例:

To be able to set Avro field to null you should allow this in Avro schema, by adding null as one of the possible types of the field. Take a look on example from Avro documentation:

{
  "type": "record",
  "name": "MyRecord",
  "fields" : [
    {"name": "userId", "type": "long"},              // mandatory field
    {"name": "userName", "type": ["null", "string"]} // optional field 
  ]
}

此处userName被声明为复合类型，可以是null或string.这种定义允许将userName字段设置为null.由于对比度userId只能包含长值，因此尝试将userId设置为null会导致NullPointerException.

here userName is declared as composite type which could be either null or string. This kind of definition allows to set userName field to null. As contrast userId can only contain long values, hence attempt to set userId to null will result in NullPointerException.

这篇关于在Avro文件中存储空值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！