如何使用Scala在spark中漂亮地打印JSON数据框? [英] How can I prettyprint a JSON Dataframe in spark with Scala?

问题描述

我有一个数据框，我想将其作为有效的json写入json文件:

I have a dataframe from that I want to write to a json file as valid json:

我当前的代码如下:

val df: DataFrame = myFun(...)
df.toJSON.saveAsTextFile( "myFile.json" )

输出格式为:

{}{}{}

如何获取文件内容以组织为有效的JSON?

How can I get the file contents to organize as valid JSON?:

[{},{},{}]

推荐答案

我使用Spray JSON的解决方法:

My workaround using Spray JSON:

def apply(df: DataFrame): Option[String] = {
    val collectedData  = df.toJSON.coalesce(1).collect().mkString("\n")
    val json = "[" + ("}\n".r replaceAllIn (collectedData, "},\n")) + "]"
    val pretty = json.parseJson.prettyPrint
    Some(s"$pretty\n")
}

丑陋且效率低下，但只要最终结果不是大数据，我就可以做，但在这种情况下，我还是不希望有一个正确的json文件.

ugly and inefficient but does what I want provided the final result isn't big-data huge, in which case I wouldn't want a single proper json file anyway.

这篇关于如何使用Scala在spark中漂亮地打印JSON数据框?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！