avro json附加字段 [英] avro json additional field

问题描述

我有以下avro模式

{
    "type":"record",
    "name":"test",
    "namespace":"test.name",
    "fields":[
        {"name":"items","type":
            {"type":"array",
                "items":
                    {"type":"record","name":"items",
                        "fields":[
                                {"name":"name","type":"string"},
                                {"name":"state","type":"string"}
                            ]
                    }
            }
        },
        {"name":"firstname","type":"string"}
    ]
}

当我使用Json解码器和avro编码器对Json数据进行编码时:

when I am using Json decoder and avro encoder to encode Json data:

val writer = new GenericDatumWriter[GenericRecord](schema)
val reader = new GenericDatumReader[GenericRecord](schema)
val baos = new ByteArrayOutputStream
val decoder: JsonDecoder = DecoderFactory.get.jsonDecoder(schema, json)
val encoder = EncoderFactory.get.binaryEncoder(baos, null)
val datum = reader.read(null, decoder)
writer.write(datum, encoder)
encoder.flush()
val avroByteArray = baos.toByteArray

场景1: 当我传递以下json进行编码时，效果很好:

scenario1: when I am passing following json to encode it works fine:

{
  "items": [
    {
      "name": "dallas",
      "state": "TX"
    }
  ],
  "firstname":"arun"
}

场景2: 当我在根级别(姓氏)在json中传递其他属性时，它可以进行编码并正常工作:

scenario2: when I am passing additional attribute in json at root level (lastname) it is able to encode and works fine:

{
  "items": [
    {
      "name": "dallas",
      "state": "TX"
    }
  ],
  "firstname":"fname",
  "lastname":"lname"
}

场景3: 当我在数组记录(国家/地区)中添加其他属性时，它会引发以下异常:

scenario3: when I am add additional attribute in array record (country) it is throwing following exception:

Expected record-end. Got FIELD_NAME
org.apache.avro.AvroTypeException: Expected record-end. Got FIELD_NAME
    at org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:698)
{
  "items": [
    {
      "name": "dallas",
      "state": "TX",
      "country":"USA"
    }
  ],
  "firstname":"fname",
  "lastname":"lname"
}

我需要使场景#3正常工作，任何帮助都会很棒.

I need to make scenario#3 working any help will be great.

推荐答案

您的架构不代表方案3中的结构:缺少国家/地区"字段:

Your schema does not represent the structure in scenario 3: the 'country' field is missing:

{"name":"country", "type":"string"}

您仅声明字段名称"和状态".然后，解码器正确地希望(sub)记录在这些记录之后结束，并且如错误消息所述，它将获得一个(另一个)字段名(国家").

You are only declaring the fields 'name' and 'state'. Then decoder correctly expects the (sub)record to end after those and, as the error message states, it gets a(nother) field name instead ('country').

顺便说一句:您可以使用生成器始终从JSON中获取匹配的模式，网络上有几个可用的

Btw: You could use a generator to always get a matching schema out of your JSON, there are a couple available in the net.

这篇关于avro json附加字段的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！