在 spark 中读取 XML [英] Read XML in spark

问题描述

我正在尝试使用 spark-xml jar 在 pyspark 中读取 xml/嵌套 xml.

I am trying to read xml/nested xml in pyspark using spark-xml jar.

df = sqlContext.read 
  .format("com.databricks.spark.xml")
   .option("rowTag", "hierachy")
   .load("test.xml"

当我执行时，数据框没有正确创建.

when I execute, data frame is not creating properly.

    +--------------------+
    |                 att|
    +--------------------+
    |[[1,Data,[Wrapped...|
    +--------------------+

我的xml格式如下:

推荐答案

heirarchy 应该是 rootTag 而 att 应该是 rowTag 作为

heirarchy should be rootTag and att should be rowTag as

df = spark.read 
    .format("com.databricks.spark.xml") 
    .option("rootTag", "hierarchy") 
    .option("rowTag", "att") 
    .load("test.xml")

你应该得到

+-----+------+----------------------------+
|Order|attval|children                    |
+-----+------+----------------------------+
|1    |Data  |[[[1, Studyval], [2, Site]]]|
|2    |Info  |[[[1, age], [2, gender]]]   |
+-----+------+----------------------------+

和架构

root
 |-- Order: long (nullable = true)
 |-- attval: string (nullable = true)
 |-- children: struct (nullable = true)
 |    |-- att: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- Order: long (nullable = true)
 |    |    |    |-- attval: string (nullable = true)

查找有关 databricks xml

这篇关于在 spark 中读取 XML的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！