如何使用 Apache Spark 计算准确的中位数? [英] How can I calculate exact median with Apache Spark?

问题描述

这个页面 包含一些统计函数(均值、标准差、方差等)但不包含中值.如何计算准确的中位数?

This page contains some statistics functions (mean, stdev, variance, etc.) but it does not contain the median. How can I calculate exact median?

推荐答案

你需要对 RDD 进行排序并取两个元素的中间或平均值.这是 RDD[Int] 的示例:

You need to sort RDD and take element in the middle or average of two elements. Here is example with RDD[Int]:

  import org.apache.spark.SparkContext._

  val rdd: RDD[Int] = ???

  val sorted = rdd.sortBy(identity).zipWithIndex().map {
    case (v, idx) => (idx, v)
  }

  val count = sorted.count()

  val median: Double = if (count % 2 == 0) {
    val l = count / 2 - 1
    val r = l + 1
    (sorted.lookup(l).head + sorted.lookup(r).head).toDouble / 2
  } else sorted.lookup(count / 2).head.toDouble

这篇关于如何使用 Apache Spark 计算准确的中位数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！