如何在Spark MLlib中为K-means初始化集群中心? [英] How to initialize cluster centers for K-means in Spark MLlib?

问题描述

在Spark MLlib中运行K-Means时是否有初始化集群中心的方法?

我尝试了以下操作:

model = KMeans.train(
    sc.parallelize(data), 3, maxIterations=0,
    initialModel = KMeansModel([(-1000.0,-1000.0),(5.0,5.0),(1000.0,1000.0)]))

spark-mllib_2.10中不存在

initialModel和setInitialModel

import org.apache.spark.mllib.clustering.{KMeans, KMeansModel} import org.apache.spark.mllib.linalg.Vectors val data = sc.parallelize(Seq( "[0.0, 0.0]", "[1.0, 1.0]", "[9.0, 8.0]", "[8.0, 9.0]" )).map(Vectors.parse(_)) val initialModel = new KMeansModel( Array("[0.6, 0.6]", "[8.0, 8.0]").map(Vectors.parse(_)) ) val model = new KMeans() .setInitialModel(initialModel) .setK(2) .run(data)

和PySpark 1.6+使用train方法的initialModel参数:

 from pyspark.mllib.clustering import KMeansModel, KMeans
from pyspark.mllib.linalg import Vectors

data = sc.parallelize([
    "[0.0, 0.0]", "[1.0, 1.0]", "[9.0, 8.0]", "[8.0,  9.0]"
]).map(Vectors.parse)

initialModel = KMeansModel([
    Vectors.parse(v) for v in ["[0.6,  0.6]", "[8.0,  8.0]"]])
model = KMeans.train(data, 2, initialModel=initialModel)
 

如果这些方法中的任何一种都不起作用，则意味着您使用的是Spark的早期版本.

Is there a way to initialize cluster centers while running K-Means in Spark MLlib?

I tried following :

model = KMeans.train(
    sc.parallelize(data), 3, maxIterations=0,
    initialModel = KMeansModel([(-1000.0,-1000.0),(5.0,5.0),(1000.0,1000.0)]))

initialModel and setInitialModel are not present in spark-mllib_2.10

解决方案

Initial model can set in Scala since Spark 1.5+ using setInitialModel which takes KMeansModel:

import org.apache.spark.mllib.clustering.{KMeans, KMeansModel}
import org.apache.spark.mllib.linalg.Vectors

val data = sc.parallelize(Seq(
    "[0.0, 0.0]", "[1.0, 1.0]", "[9.0, 8.0]", "[8.0,  9.0]"
)).map(Vectors.parse(_))

val initialModel = new KMeansModel(
   Array("[0.6,  0.6]", "[8.0,  8.0]").map(Vectors.parse(_))
)

val model = new KMeans()
  .setInitialModel(initialModel)
  .setK(2)
  .run(data)

and PySpark 1.6+ using initialModel parameter to train method:

from pyspark.mllib.clustering import KMeansModel, KMeans
from pyspark.mllib.linalg import Vectors

data = sc.parallelize([
    "[0.0, 0.0]", "[1.0, 1.0]", "[9.0, 8.0]", "[8.0,  9.0]"
]).map(Vectors.parse)

initialModel = KMeansModel([
    Vectors.parse(v) for v in ["[0.6,  0.6]", "[8.0,  8.0]"]])
model = KMeans.train(data, 2, initialModel=initialModel)

If any of these methods doesn't work it means that you're using an earlier version of Spark.

这篇关于如何在Spark MLlib中为K-means初始化集群中心?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！