将 Spark org.apache.spark.mllib.linalg.Matrix 保存到文件 [英] Save Spark org.apache.spark.mllib.linalg.Matrix to a file
本文介绍了将 Spark org.apache.spark.mllib.linalg.Matrix 保存到文件的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
Spark MLLib 中的相关结果是 org.apache.spark.mllib.linalg.Matrix 类型.(参见 http://spark.apache.org/docs/1.2.1/mllib-statistics.html#correlations)
The result of correlation in Spark MLLib is a of type org.apache.spark.mllib.linalg.Matrix. (see http://spark.apache.org/docs/1.2.1/mllib-statistics.html#correlations)
val data: RDD[Vector] = ...
val correlMatrix: Matrix = Statistics.corr(data, "pearson")
我想将结果保存到文件中.我该怎么做?
I would like to save the result into a file. How can I do this?
推荐答案
这里有一个简单有效的方法,可以将 Matrix 保存到 hdfs 并指定分隔符.
Here is a simple and effective approach to save the Matrix to hdfs and specify the separator.
(使用转置是因为 .toArray 是列主要格式.)
(The transpose is used since .toArray is in column major format.)
val localMatrix: List[Array[Double]] = correlMatrix
.transpose // Transpose since .toArray is column major
.toArray
.grouped(correlMatrix.numCols)
.toList
val lines: List[String] = localMatrix
.map(line => line.mkString(" "))
sc.parallelize(lines)
.repartition(1)
.saveAsTextFile("hdfs:///home/user/spark/correlMatrix.txt")
这篇关于将 Spark org.apache.spark.mllib.linalg.Matrix 保存到文件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文