减火花org.apache.spark.mllib.linalg.Matrix到文件 [英] Save Spark org.apache.spark.mllib.linalg.Matrix to a file

问题描述

在火花MLLib相关的结果是一式org.apache.spark.mllib.linalg.Matrix的。 （请参见 http://spark.apache.org/docs/1.2 .1 / mllib-statistics.html＃相关）

The result of correlation in Spark MLLib is a of type org.apache.spark.mllib.linalg.Matrix. (see http://spark.apache.org/docs/1.2.1/mllib-statistics.html#correlations)

val data: RDD[Vector] = ... 

val correlMatrix: Matrix = Statistics.corr(data, "pearson")

我想结果保存到一个文件中。我怎样才能做到这一点？

I would like to save the result into a file. How can I do this?

推荐答案

下面是一个简单而有效的方法，以矩阵保存到HDFS并指定分隔符。

Here is a simple and effective approach to save the Matrix to hdfs and specify the separator.

（因为.toArray使用转置是在列主要格式。）

(The transpose is used since .toArray is in column major format.)

val localMatrix: List[Array[Double]] = correlMatrix
    .transpose  // Transpose since .toArray is column major
    .toArray
    .grouped(correlMatrix.numCols)
    .toList

val lines: List[String] = localMatrix
    .map(line => line.mkString(" "))

sc.parallelize(lines)
    .repartition(1)
    .saveAsTextFile("hdfs:///home/user/spark/correlMatrix.txt")

这篇关于减火花org.apache.spark.mllib.linalg.Matrix到文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！