如何从 Scala 的可迭代列表创建 DataFrame? [英] How to create DataFrame from Scala's List of Iterables?

问题描述

我有以下 Scala 值:

val values: List[Iterable[Any]] = Traces().evaluate(features).toList

我想将其转换为 DataFrame.

当我尝试以下操作时:

sqlContext.createDataFrame(values)

我收到此错误:

 错误:重载方法值 createDataFrame 与替代:[A <: Product](data: Seq[A])(隐式证据$2:reflect.runtime.universe.TypeTag[A])org.apache.spark.sql.DataFrame[A <: Product](rdd: org.apache.spark.rdd.RDD[A])(隐式证据$1:reflect.runtime.universe.TypeTag[A])org.apache.spark.sql.DataFrame不能应用于 (List[Iterable[Any]])sqlContext.createDataFrame(values)

为什么?

解决方案

正如 zero323 提到的，我们需要先转换List[Iterable[Any]] 到 List[Row] 然后将行放入 RDD 并为 spark 数据帧准备模式.>

将List[Iterable[Any]]转换为List[Row]，我们可以说

val rows = values.map{x =>行(x:_*)}

然后有了像schema这样的模式，我们就可以制作RDD

val rdd = sparkContext.makeRDD[RDD](rows)

最后创建一个spark数据框

val df = sqlContext.createDataFrame(rdd, schema)

I have the following Scala value:

val values: List[Iterable[Any]] = Traces().evaluate(features).toList

and I want to convert it to a DataFrame.

When I try the following:

sqlContext.createDataFrame(values)

I got this error:

error: overloaded method value createDataFrame with alternatives:

[A <: Product](data: Seq[A])(implicit evidence$2: reflect.runtime.universe.TypeTag[A])org.apache.spark.sql.DataFrame 
[A <: Product](rdd: org.apache.spark.rdd.RDD[A])(implicit evidence$1: reflect.runtime.universe.TypeTag[A])org.apache.spark.sql.DataFrame
cannot be applied to (List[Iterable[Any]])
          sqlContext.createDataFrame(values)

Why?

解决方案

As zero323 mentioned, we need to first convert List[Iterable[Any]] to List[Row] and then put rows in RDD and prepare schema for the spark data frame.

To convert List[Iterable[Any]] to List[Row], we can say

val rows = values.map{x => Row(x:_*)}

and then having schema like schema, we can make RDD

val rdd = sparkContext.makeRDD[RDD](rows)

and finally create a spark data frame

val df = sqlContext.createDataFrame(rdd, schema)

这篇关于如何从 Scala 的可迭代列表创建 DataFrame?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！