如何计算与Apache星火精确的中间? [英] How can I calculate exact median with Apache Spark?
本文介绍了如何计算与Apache星火精确的中间?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
<一个href=\"https://spark.apache.org/docs/0.7.0/api/core/spark/api/java/JavaDoubleRDD.html\">https://spark.apache.org/docs/0.7.0/api/core/spark/api/java/JavaDoubleRDD.html
本页面包含了一些统计功能(平均值,标准偏差,方差等),但它不包含中位数。我怎样才能精确计算中位数?
This page contains some statistics functions(mean, stdev, variance etc.) but it does not contains median. How can I calculate exact median ?
感谢
推荐答案
您需要梳理RDD并采取元素的两个元素的中间或平均值。这里是例如使用RDD [INT]:
You need to sort RDD and take element in the middle or average of two elements. Here is example with RDD[Int]:
import org.apache.spark.SparkContext._
val rdd: RDD[Int] = ???
val sorted = rdd.sortBy(identity).zipWithIndex().map {
case (v, idx) => (idx, v)
}
val count = sorted.count()
val median: Double = if (count % 2 == 0) {
val l = count / 2 - 1
val r = l + 1
(sorted.lookup(l).head + sorted.lookup(r).head).toDouble / 2
} else sorted.lookup(count / 2).head.toDouble
这篇关于如何计算与Apache星火精确的中间?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文