转换矩阵,以RowMatrix在使用Scala的Apache星火 [英] Convert Matrix to RowMatrix in Apache Spark using Scala
问题描述
我真的很想我的org.apache.spark.mllib.linalg.Matrix转换为org.apache.spark.mllib.linalg.distributed.RowMatrix
I'd really like to convert my org.apache.spark.mllib.linalg.Matrix to org.apache.spark.mllib.linalg.distributed.RowMatrix
我能做到这样:
val xx = X.computeGramianMatrix() //xx is type org.apache.spark.mllib.linalg.Matrix
val xxs = xx.toString()
val xxr = xxs.split("\n").map(row => row.replace(" "," ").replace(" "," ").replace(" "," ").replace(" "," ").replace(" ",",").split(","))
val xxp = sc.parallelize(xxr)
val xxd = xxp.map(ar => Vectors.dense(ar.map(elm => elm.toDouble)))
val xxrm: RowMatrix = new RowMatrix(xxd)
不过,这确实是粗暴和总的黑客。有人能告诉我一个更好的办法?
However, that is really gross and a total hack. Can someone show me a better way?
请注意我用的星火版本1.3.0
Note I am using Spark version 1.3.0
推荐答案
我建议您将矩阵转换为RDD [矢量],它可以自动转换为RowMatrix。
I suggest that you convert your Matrix to an RDD[Vector] which you can automatically convert to a RowMatrix.
让我们看看下面的例子:
Let's consider the following example :
import org.apache.spark.rdd._
import org.apache.spark.mllib.linalg._
val denseData = Seq(
Vectors.dense(0.0, 1.0, 2.0),
Vectors.dense(3.0, 4.0, 5.0),
Vectors.dense(6.0, 7.0, 8.0),
Vectors.dense(9.0, 0.0, 1.0)
)
val dm: Matrix = Matrices.dense(3, 2, Array(1.0, 3.0, 5.0, 2.0, 4.0, 6.0))
您需要定义一个方法,你的矩阵转换为RDD [矢量]
You'll need to define a method to convert your Matrix to an RDD[Vector]
def matrixToRDD(m: Matrix): RDD[Vector] = {
val columns = m.toArray.grouped(m.numRows)
val rows = columns.toSeq.transpose // Skip this if you want a column-major RDD.
val vectors = rows.map(row => new DenseVector(row.toArray))
sc.parallelize(vectors)
}
,现在你可以适用于你的矩阵转换:
and now you can apply the conversion on your Matrix :
import org.apache.spark.mllib.linalg.distributed.RowMatrix
val rows = matrixToRDD(dm)
val mat = new RowMatrix(rows)
我希望这可以帮助!
I hope that this can help!
这篇关于转换矩阵,以RowMatrix在使用Scala的Apache星火的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!