星火拼花嵌套值压扁 [英] Spark parquet nested value flatten

问题描述

我有拼花文件。我装使用Spark.And价值的一个关键嵌套，值对。我如何扁平化？

I have parquet file. I loaded using Spark.And one of the value is nested key,value pairs. How do I flatten?

df.printSchema
root
|-- location: string (nullable = true)
|-- properties: string (nullable = true)


texas,{"key":{"key1":"value1","key2":"value2"}}

感谢，

推荐答案

您可以使用<一个href=\"https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/sql/DataFrame.html#explode(java.lang.String,%20java.lang.String,%20scala.Function1,%20scala.reflect.api.TypeTags.TypeTag)\"相对=nofollow>爆炸你的数据帧，并把它传递函数读取使用scala4s的JSON列。 Scala4s方便解析API，为你的情况下，它看起来像：

You can use explode on your dataframe and pass it a function that reads the JSON column using scala4s. Scala4s has easy parsing API, for your case it will look like:

val list = for {
  JArray(keys) <- parse(json) \\ "key"
  json @ JObject(key) <- keys
  JField("key1", JString(key1)) <- key
  JField("key2", JString(key2)) <- key
} yield {
  Seq(key1, key2)
}

这平坦的数据帧。

如果您也想为键添加列，你可以使用<一个href=\"https://spark.apache.org/docs/1.6.1/api/java/org/apache/spark/sql/DataFrame.html#withColumn(java.lang.String,%20org.apache.spark.sql.Column)\"相对=nofollow> withColumn 后爆炸（保持键也新列）。

If you also want to add column for key, you can use withColumn after explode(keep the key also in the new column).

这篇关于星火拼花嵌套值压扁的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！