为什么Spark Mllib KMeans算法非常慢? [英] Why is Spark Mllib KMeans algorithm extremely slow?

问题描述

我遇到与此帖子相同的问题，但是我的积分不足，无法在此处添加评论.我的数据集有100万行，100个列.我也在使用Mllib KMeans，它非常慢.实际上，这项工作永远不会完成，我必须杀死它.我正在Google云(dataproc)上运行它.如果我要求的集群数量较少(k = 1000)，它将运行，但是仍然需要35分钟以上的时间.我需要它来运行k〜5000.我不知道为什么这么慢.给定工作器/节点的数量以及在100万x〜300,000 col矩阵上的SVD大约需要3分钟，才能对数据进行适当的分区，但是当涉及到KMeans时，它只是陷入了一个黑洞.我现在尝试的迭代次数较少(从2次而不是100次)，但是我觉得某个地方出了问题.

I'm having the same problem as in this post, but I don't have enough points to add a comment there. My dataset has 1 Million rows, 100 cols. I'm using Mllib KMeans also and it is extremely slow. The job never finishes in fact and I have to kill it. I am running this on Google cloud (dataproc). It runs if I ask for a smaller number of clusters (k=1000), but still take more than 35 minutes. I need it to run for k~5000. I have no idea why is it so slow. The data is properly partitioned given the number of workers/nodes and SVD on a 1 million x ~300,000 col matrix takes ~3 minutes, but when it comes to KMeans it just goes into a black hole. I am now trying a lower number of iterations (2 instead of 100), but I feel something is wrong somewhere.

KMeansModel Cs = KMeans.train(datamatrix, k, 100);//100 iteration, changed to 2 now. # of clusters k=1000 or 5000

推荐答案

原因似乎相对简单.您使用非常大的k并将其与昂贵的初始化算法结合在一起.

It looks like the reason is relatively simple. You use quite large k and combine it with an expensive initialization algorithm.

默认情况下，Spark使用 K-means ++ 称为K-均值|| (请参见 Spark MLLib中的Kmeans ++中的initializeSteps参数到底是什么?).分布式版本大致为 O(k)，因此，如果k较大，则启动速度可能会较慢.这应该可以解释为什么减少迭代次数后却看不到任何改善.

By default Spark is using as distributed variant of K-means++ called K-means|| (see What exactly is the initializationSteps parameter in Kmeans++ in Spark MLLib?). Distributed version is roughly O(k) so with larger k you can expect slower start. This should explain why you see no improvement when you reduce number of iterations.

训练模型时，使用大K也很昂贵. Spark使用的是劳埃德(Lloyds)的变体，大致为 O(nkdi).

Using large K is also expensive when model is trained. Spark is using a variant of Lloyds which is roughly O(nkdi).

如果您期望数据的复杂结构，那么最有可能使用一种比K-Means更好的算法来进行处理，但是，如果您真的想坚持下去，则可以使用随机初始化.

If you expect complex structure of the data there most likely a better algorithms out there to handle this than K-Means but if you really want to stick with it you start with using random initialization.

这篇关于为什么Spark Mllib KMeans算法非常慢?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！