Json 字段在转换为 spark DataFrame 时默认排序 [英] Json fields getting sorted by default when converted to spark DataFrame

问题描述

当我从 json 文件创建数据帧时，json 文件中的字段默认在数据帧中排序.如何避免这种排序?
Jsonfile 每行有一个 json 消息:

When I create a dataframe from json file, the fields from the json file are sorted by default in the dataframe. How to avoid this sorting?
Jsonfile having one json message per line:

{"name":"john","age":10,"class":2}{"name":"rambo","age":11,"class":3}

当我从这个文件创建数据框时:
val jDF = sqlContext.read.json("/user/inputfiles/sample.json")

When I create Data frame from this file as:
val jDF = sqlContext.read.json("/user/inputfiles/sample.json")

一个DF被创建为
jDF: org.apache.spark.sql.DataFrame = [age: bigint, class: bigint, name: string].在 DF 中，字段默认排序.
我们如何避免这种情况发生?

a DF is created as
jDF: org.apache.spark.sql.DataFrame = [age: bigint, class: bigint, name: string] . In the DF the fields are sorted by default.
How do we avoid this from happening?

我无法理解这里出了什么问题.

Im unable to understand what is going wrong here.

感谢您对解决问题的任何帮助.

Appreciate any help in sorting out the problem.

推荐答案

对于问题 1:

一个简单的方法是在DataFrame上做select:

A simple way is to do select on the DataFrame:

val newDF = jDF.select("name","age","class")

参数的顺序就是你想要的列的顺序.

The order of parameters is the order of the columns you want.

但是如果有很多列并且您必须自己定义顺序，这可能会很冗长.

But this could be verbose if there are many columns and you have to define the order yourself.

这篇关于Json 字段在转换为 spark DataFrame 时默认排序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！