Spark Scala-如何从嵌套JSON构造Scala Map? [英] Spark Scala - How to construct Scala Map from nested JSON?

问题描述

我有一个嵌套的json数据，其中包含要提取和构造Scala Map的嵌套字段.

I've a nested json data with nested fields that I want to extract and construct a Scala Map.

这里有示例JSON:

"nested_field": [
  {
    "airport": "sfo",
    "score": 1.0
  },
  {
    "airport": "phx",
    "score": 1.0
  },
  {
    "airport": "sjc",
    "score": 1.0
  }
]

我想使用 saveToES ()并构建一个Scala Map，以通过以下方式将字段索引到ES索引中:

I want to use saveToES() and construct a Scala Map to index the field into ES index with mapping as below:

 "nested_field": {
    "properties": {
      "score": {
        "type": "double"
      },
      "airport": {
        "type": "keyword",
        "ignore_above": 1024
      }
    }
  }

使用spark.read.json("example.json")将json文件读入数据框.在这种情况下构造Scala Map的正确方法是什么?

The json file is read into the dataframe using spark.read.json("example.json"). Whats the right way to construct the Scala Map in this case?

感谢您的帮助！

推荐答案

您可以使用下面的示例代码来实现

You can do it using the below sample code

import org.json4s.DefaultFormats
import org.json4s.jackson.JsonMethods.parse



  case class AirPortScores(airport: String, score: Double)
  case class JsonRulesHandler(airports: List[AirPortScores])

  val jsonString: String = """{"airports":[{"airport":"sfo","score":1},{"airport":"phx","score":1},{"airport":"sjc","score":1}]}"""

  def loadJsonString(JsonString: String): JsonRulesHandler = {
  implicit val formats: DefaultFormats.type = org.json4s.DefaultFormats
  parse(JsonString).extract[JsonRulesHandler]
}

val parsedJson: JsonRulesHandler = loadJsonString(jsonString)
parsedJson.airports.foreach(println)//you can select parsedJson.airport or scores
//below ouput
AirPortScores(sfo,1.0)
AirPortScores(phx,1.0)
AirPortScores(sjc,1.0)

这篇关于Spark Scala-如何从嵌套JSON构造Scala Map?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！