从JSON文档生成AVRO模式 [英] generating an AVRO schema from a JSON document
问题描述
是否有任何工具可以从典型" JSON文档创建AVRO模式.
Is there any tool able to create an AVRO schema from a 'typical' JSON document.
例如:
{
"records":[{"name":"X1","age":2},{"name":"X2","age":4}]
}
我发现 http://jsonschema.net/reboot/#/会生成" json模式" '
{
"$schema": "http://json-schema.org/draft-04/schema#",
"id": "http://jsonschema.net#",
"type": "object",
"required": false,
"properties": {
"records": {
"id": "#records",
"type": "array",
"required": false,
"items": {
"id": "#1",
"type": "object",
"required": false,
"properties": {
"name": {
"id": "#name",
"type": "string",
"required": false
},
"age": {
"id": "#age",
"type": "integer",
"required": false
}
}
}
}
}
}
但是我想要一个 AVRO 版本.
推荐答案
您可以使用Apache Spark和python轻松实现这一目标.首先从 http://spark.apache.org/downloads.html 下载spark发行版,然后使用pip
为python安装avro
软件包.然后使用avro软件包运行pyspark:
You can achieve that easily using Apache Spark and python. First download the spark distribution from http://spark.apache.org/downloads.html, then install avro
package for python using pip
. Then run pyspark with avro package:
./bin/pyspark --packages com.databricks:spark-avro_2.11:3.1.0
并使用以下代码(假设input.json
文件包含一个或多个json文档,每个文档位于单独的行中):
and use the following code (assuming the input.json
files contains one or more json documents, each in separate line):
import os, avro.datafile
spark.read.json('input.json').coalesce(1).write.format("com.databricks.spark.avro").save("output.avro")
avrofile = filter(lambda file: file.startswith('part-r-00000'), os.listdir('output.avro'))[0]
with open('output.avro/' + avrofile) as avrofile:
reader = avro.datafile.DataFileReader(avrofile, avro.io.DatumReader())
print(reader.datum_reader.writers_schema)
例如:对于具有内容的输入文件:
For example: for input file with content:
{'string': 'somestring', 'number': 3.14, 'structure': {'integer': 13}}
{'string': 'somestring2', 'structure': {'integer': 14}}
该脚本将导致:
{"fields": [{"type": ["double", "null"], "name": "number"}, {"type": ["string", "null"], "name": "string"}, {"type": [{"type": "record", "namespace": "", "name": "structure", "fields": [{"type": ["long", "null"], "name": "integer"}]}, "null"], "name": "structure"}], "type": "record", "name": "topLevelRecord"}
这篇关于从JSON文档生成AVRO模式的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!