在星火简单的矩阵乘法 [英] simple matrix multiplication in Spark

问题描述

我有一些非常基本的火花code挣扎。我想定义一个矩阵 X 2列。这是我曾尝试：

 斯卡拉＆GT; VAL S = breeze.linalg.linspace（-3,3,5-三）
小号：breeze.linalg.DenseVector [双] = DenseVector（-3.0，-1.5，0.0，1.5，3.0）//在这种情况下，我想s到既列1和x 2列斯卡拉＆GT; VAL SS = s.toArray ++ s.toArray
SS：数组[双] =阵列（-3.0，-1.5，0.0，1.5，3.0，-3.0，-1.5，0.0，1.5，3.0）斯卡拉＆GT;进口org.apache.spark.mllib.linalg.distributed.RowMatrix
进口org.apache.spark.mllib.linalg.distributed.RowMatrix斯卡拉＆GT; VAL垫=新RowMatrix（SS，5，2）
＆LT;＆控制台GT;：17：错误：类型不匹配;
 发现：数组[双]
 要求：org.apache.spark.rdd.RDD [org.apache.spark.mllib.linalg.Vector]
       VAL垫=新RowMatrix（SS，5，2）
 

我不明白我怎么能得到正确的改造，以便将值传递到分布式矩阵^

编辑：
也许我已经能够解决：

 斯卡拉＆GT; VAL S = breeze.linalg.linspace（-3,3,5-三）
小号：breeze.linalg.DenseVector [双] = DenseVector（-3.0，-1.5，0.0，1.5，3.0）斯卡拉＆GT; VAL SS = s.to
的toArray toDenseMatrix toDenseVector toScalaVector的toString
toVector斯卡拉＆GT; VAL SS = s.toArray ++ s.toArray
SS：数组[双] =阵列（-3.0，-1.5，0.0，1.5，3.0，-3.0，-1.5，0.0，1.5，3.0）斯卡拉＆GT; VAL X =新breeze.linalg.Dense
DenseMatrix DenseVector斯卡拉＆GT; VAL X =新breeze.linalg.DenseMatrix（5,2，SS）
X：breeze.linalg.DenseMatrix [双] =
-3.0 -3.0
-1.5 -1.5
0.0 0.0
1.5 1.5
3.0 3.0斯卡拉＆GT; VAL xDist = sc.parallelize（x.toArray）
xDist：org.apache.spark.rdd.RDD [双] = ParallelCollectionRDD [0]并行化AT＆LT;＆控制台GT;：18
 

解决方案

这样的事情。这typechecks，但由于某种原因不能在我的斯卡拉工作运行。

 进口org.apache.spark.mllib.linalg._
进口org.apache.spark.mllib.linalg.distributed._
进口org.apache.spark.SparkContext
进口org.apache.spark.SparkContext._
进口org.apache.spark.SparkConf
进口org.apache.spark.rdd.RDDVAL的conf =新SparkConf（）。setAppName（火花从零开始）。setMaster（本地）
VAL SC =新SparkContext（CONF）//值的每一行中的列
VAL COL =名单（-3.0，-1.5，0.0，1.5，3.0）;//使列值的两排，转吧，
//使结果的矢量
VAL T =名单（COL，COL）.transpose.map（R = GT; Vectors.dense（r.toArray））//载体的产生顺序做出RDD，和
//使从一个RowMatrix。
VAL RM =新RowMatrix（sc.makeRDD（T））;
 

I am struggling with some very basic spark code. I would like to define a matrix x with 2 columns. This is what I have tried:

scala> val s = breeze.linalg.linspace(-3,3,5)
s: breeze.linalg.DenseVector[Double] = DenseVector(-3.0, -1.5, 0.0, 1.5, 3.0) // in this case I want s to be both column 1 and column 2 of x

scala> val ss = s.toArray ++ s.toArray
ss: Array[Double] = Array(-3.0, -1.5, 0.0, 1.5, 3.0, -3.0, -1.5, 0.0, 1.5, 3.0)

scala> import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.mllib.linalg.distributed.RowMatrix

scala> val mat = new RowMatrix(ss, 5, 2)
<console>:17: error: type mismatch;
 found   : Array[Double]
 required: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector]
       val mat = new RowMatrix(ss, 5, 2)

I do not understand how I can get the right transformation in order to pass the values to the distributed matrix ^

EDIT: Maybe I have been able to solve:

scala> val s = breeze.linalg.linspace(-3,3,5)
s: breeze.linalg.DenseVector[Double] = DenseVector(-3.0, -1.5, 0.0, 1.5, 3.0)

scala> val ss = s.to
toArray         toDenseMatrix   toDenseVector   toScalaVector   toString        
toVector        

scala> val ss = s.toArray ++ s.toArray
ss: Array[Double] = Array(-3.0, -1.5, 0.0, 1.5, 3.0, -3.0, -1.5, 0.0, 1.5, 3.0)

scala> val x = new breeze.linalg.Dense
DenseMatrix   DenseVector   

scala> val x = new breeze.linalg.DenseMatrix(5, 2, ss)
x: breeze.linalg.DenseMatrix[Double] = 
-3.0  -3.0  
-1.5  -1.5  
0.0   0.0   
1.5   1.5   
3.0   3.0   

scala> val xDist = sc.parallelize(x.toArray)
xDist: org.apache.spark.rdd.RDD[Double] = ParallelCollectionRDD[0] at parallelize at <console>:18

解决方案

Something like this. This typechecks, but for some reason won't run in my Scala worksheet.

import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.linalg.distributed._
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.rdd.RDD

val conf = new SparkConf().setAppName("spark-scratch").setMaster("local")
val sc= new SparkContext(conf)

// the values for the column in each row
val col = List(-3.0, -1.5, 0.0, 1.5, 3.0) ;

// make two rows of the column values, transpose it,
// make Vectors of the result
val t = List(col,col).transpose.map(r=>Vectors.dense(r.toArray))

// make an RDD from the resultant sequence of Vectors, and 
// make a RowMatrix from that.
val rm = new RowMatrix(sc.makeRDD(t));

这篇关于在星火简单的矩阵乘法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！