我应该如何将 org.apache.spark.ml.linalg.Vector 的 RDD 转换为数据集? [英] How should I convert an RDD of org.apache.spark.ml.linalg.Vector to Dataset?

问题描述

我正在努力理解 RDD、DataSet 和 DataFrame 之间的转换是如何工作的.我对 Spark 很陌生，每次我需要从数据模型传递到另一个数据模型(尤其是从 RDD 到数据集和数据帧)时，我都会卡住.谁能给我解释一下正确的做法?

I'm struggling to understand how the conversion among RDDs, DataSets and DataFrames works. I'm pretty new to Spark, and I get stuck every time I need to pass from a data model to another (especially from RDDs to Datasets and Dataframes). Could anyone explain me the right way to do it?

举个例子，现在我有一个 RDD[org.apache.spark.ml.linalg.Vector] 并且我需要将它传递给我的机器学习算法，例如一个 KMeans (Spark数据集 MLlib).因此，我需要将其转换为 Dataset，其中有一列名为features"的列应包含 Vector 类型的行.我该怎么做?

As an example, now I have a RDD[org.apache.spark.ml.linalg.Vector] and I need to pass it to my machine learning algorithm, for example a KMeans (Spark DataSet MLlib). So, I need to convert it to Dataset with a single column named "features" which should contain Vector typed rows. How should I do this?

推荐答案

要将 RDD 转换为 dataframe，最简单的方法是在 Scala 中使用 toDF().要使用此功能，必须导入使用 SparkSession 对象完成的隐式.可以按如下方式进行:

To convert a RDD to a dataframe, the easiest way is to use toDF() in Scala. To use this function, it is necessary to import implicits which is done using the SparkSession object. It can be done as follows:

val spark = SparkSession.builder().getOrCreate()
import spark.implicits._

val df = rdd.toDF("features")

toDF() 采用元组的 RDD.当 RDD 由常见的 Scala 对象组成时，它们将被隐式转换，即不需要做任何事情，当 RDD 有多个列时也不需要做任何事情，RDD 已经包含一个元组.但是，在这种特殊情况中，您需要先将 RDD[org.apache.spark.ml.linalg.Vector] 转换为 RDD[(org.apache.spark.ml.linalg.Vector)].因此，需要对元组做如下转换:

toDF() takes an RDD of tuples. When the RDD is built up of common Scala objects they will be implicitly converted, i.e. there is no need to do anything, and when the RDD has multiple columns there is no need to do anything either, the RDD already contains a tuple. However, in this special case you need to first convert RDD[org.apache.spark.ml.linalg.Vector] to RDD[(org.apache.spark.ml.linalg.Vector)]. Therefore, it is necessary to do a convertion to tuple as follows:

val df = rdd.map(Tuple1(_)).toDF("features")

上面的代码会将 RDD 转换为一个数据框，其中只有一个名为 features 的列.

The above will convert the RDD to a dataframe with a single column called features.

要转换为数据集，最简单的方法是使用案例类.确保案例类是在 Main 对象之外定义的.首先将 RDD 转换为数据帧，然后执行以下操作:

To convert to a dataset the easiest way is to use a case class. Make sure the case class is defined outside the Main object. First convert the RDD to a dataframe, then do the following:

case class A(features: org.apache.spark.ml.linalg.Vector)

val ds = df.as[A]

要显示所有可能的转换，可以使用 .rdd 从数据帧或数据集访问底层 RDD:

To show all possible convertions, to access the underlying RDD from a dataframe or dataset can be done using .rdd:

val rdd = df.rdd

<小时>

与在 RDD 和数据帧/数据集之间来回转换不同，使用数据帧 API 进行所有计算通常更容易.如果没有合适的函数来做你想做的事，通常可以定义一个 UDF，用户定义的函数.参见此处的示例:https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-udfs.html

这篇关于我应该如何将 org.apache.spark.ml.linalg.Vector 的 RDD 转换为数据集?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！