如何使用案例类将简单的DataFrame转换为DataSet Spark Scala? [英] How to convert a simple DataFrame to a DataSet Spark Scala with case class?

问题描述

我试图从Spark中的示例将简单的DataFrame转换为DataSet: https://spark.apache.org/docs/latest/sql- programming-guide.html

I am trying to convert a simple DataFrame to a DataSet from the example in Spark: https://spark.apache.org/docs/latest/sql-programming-guide.html

case class Person(name: String, age: Int)    
import spark.implicits._

val path = "examples/src/main/resources/people.json"

val peopleDS = spark.read.json(path).as[Person]
peopleDS.show()

但是出现了以下问题:

Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot up cast `age` from bigint to int as it may truncate
The type path of the target object is:
- field (class: "scala.Int", name: "age")
- root class: ....

有人可以帮我吗?

编辑 我注意到用Long代替Int可以工作！ 为什么会这样?

Edit I noticed that with Long instead of Int works! Why is that?

也:

val primitiveDS = Seq(1,2,3).toDS()
val augmentedDS = primitiveDS.map(i => ("var_" + i.toString, (i + 1).toLong))
augmentedDS.show()

augmentedDS.as[Person].show()

打印:

+-----+---+
|   _1| _2|
+-----+---+
|var_1|  2|
|var_2|  3|
|var_3|  4|
+-----+---+

Exception in thread "main"
org.apache.spark.sql.AnalysisException: cannot resolve '`name`' given input columns: [_1, _2];

有人可以帮我理解吗?

推荐答案

如果将Int更改为Long(或BigInt)，则效果很好:

If you change Int to Long (or BigInt) it works fine:

case class Person(name: String, age: Long)
import spark.implicits._

val path = "examples/src/main/resources/people.json"

val peopleDS = spark.read.json(path).as[Person]
peopleDS.show()

输出:

+----+-------+
| age|   name|
+----+-------+
|null|Michael|
|  30|   Andy|
|  19| Justin|
+----+-------+

默认情况下，Spark.read.json将数字解析为Long类型-这样做更安全. 您可以在使用Cast或udfs之后更改col类型.

Spark.read.json by default parses numbers as Long types - it's safer to do so. You can change the col type after using casting or udfs.

要回答第二个问题，您需要正确命名各列，然后才能转换为Person:

To answer your 2nd question, you need to name the columns correctly before the conversion to Person will work:

val primitiveDS = Seq(1,2,3).toDS()
val augmentedDS = primitiveDS.map(i => ("var_" + i.toString, (i + 1).toLong)).
 withColumnRenamed ("_1", "name" ).
 withColumnRenamed ("_2", "age" )
augmentedDS.as[Person].show()

输出:

+-----+---+
| name|age|
+-----+---+
|var_1|  2|
|var_2|  3|
|var_3|  4|
+-----+---+

这篇关于如何使用案例类将简单的DataFrame转换为DataSet Spark Scala?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！