如何在Spark SQL中找到分组的Vector列的均值? [英] How to find mean of grouped Vector columns in Spark SQL?

本文介绍了如何在Spark SQL中找到分组的Vector列的均值?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我通过调用 instances.groupBy(instances.col( property_name))创建了 RelationalGroupedDataset

val x = instances.groupBy(instances.col("property_name"))

如何撰写用户定义的汇总函数以执行 Statistics.colStats()。mean 吗?

谢谢!

推荐答案

火花> = 2.4

您可以使用摘要器

import org.apache.spark.ml.stat.Summarizer

val dfNew = df.as[(Int, org.apache.spark.mllib.linalg.Vector)]
  .map { case (group, v) => (group, v.asML) }
  .toDF("group", "features")


dfNew
  .groupBy($"group")
  .agg(Summarizer.mean($"features").alias("means"))
  .show(false)



+-----+--------------------------------------------------------------------+
|group|means                                                               |
+-----+--------------------------------------------------------------------+
|1    |[8.740630742016827E12,2.6124956666260462E14,3.268714653521495E14]   |
|6    |[2.1153266920139112E15,2.07232483974322592E17,6.2715161747245427E17]|
|3    |[6.3781865566442836E13,8.359124419656149E15,1.865567821598214E14]   |
|5    |[4.270201403521642E13,6.561211706745676E13,8.395448246737938E15]    |
|9    |[3.577032684241448E16,2.5432362841314468E16,2.3744826986293008E17]  |
|4    |[2.339253775419023E14,8.517531902022505E13,3.055115780965264E15]    |
|8    |[8.029924756674456E15,7.284873600992855E17,3.08621303029924E15]     |
|7    |[3.2275104122699105E15,7.5472363442090208E16,7.022556624056291E14]  |
|10   |[1.2412562261010224E16,5.741115713769269E15,4.34336779990902E16]    |
|2    |[1.085528901765636E16,7.633370115869126E12,6.952642232477029E11]    |
+-----+--------------------------------------------------------------------+

火花< 2.4

您不能使用 UserDefinedAggregateFunction ,但可以创建 Aggregator 使用相同的 MultivariateOnlineSummarizer

You cannot use UserDefinedAggregateFunction but you can create an Aggregator using the same MultivariateOnlineSummarizer:

import org.apache.spark.sql.Row
import org.apache.spark.sql.expressions.Aggregator
import org.apache.spark.mllib.linalg.{Vector, Vectors}
import org.apache.spark.sql.{Encoder, Encoders}
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.mllib.stat.MultivariateOnlineSummarizer

type Summarizer = MultivariateOnlineSummarizer

case class VectorSumarizer(f: String) extends Aggregator[Row, Summarizer, Vector]
    with Serializable {
  def zero = new Summarizer
  def reduce(acc: Summarizer, x: Row) = acc.add(x.getAs[Vector](f))
  def merge(acc1: Summarizer, acc2: Summarizer) = acc1.merge(acc2)

  // This can be easily generalized to support additional statistics
  def finish(acc: Summarizer) = acc.mean

  def bufferEncoder: Encoder[Summarizer] = Encoders.kryo[Summarizer]
  def outputEncoder: Encoder[Vector] = ExpressionEncoder()
}

示例用法:

import org.apache.spark.mllib.random.RandomRDDs.logNormalVectorRDD

val df = spark.sparkContext.union((1 to 10).map(i => 
  logNormalVectorRDD(spark.sparkContext, i, 10, 10000, 3, 1).map((i, _))
)).toDF("group", "features")

df
 .groupBy($"group")
 .agg(VectorSumarizer("features").toColumn.alias("means"))
 .show(10, false)

结果:

+-----+---------------------------------------------------------------------+
|group|means                                                                |
+-----+---------------------------------------------------------------------+
|1    |[1.0495089547176625E15,3.057434217141363E13,8.180842267228103E13]    |
|6    |[8.578684690153061E15,1.865830977115807E14,1.0690831496167929E15]    |
|3    |[1.0347016972600206E14,4.952536828257269E15,8.498944924018858E13]    |
|5    |[2.2135916061736424E16,1.5137112888230388E14,8.154750681129871E14]   |
|9    |[6.496030194110956E15,6.2697260327708368E16,3.7282521260607136E16]   |
|4    |[2.4518629692233766E14,1.959083619621557E13,5.278689364420169E13]    |
|8    |[1.806052212008392E16,2.0410654639336184E16,6.409495244104527E15]    |
|7    |[1.32896092658714784E17,1.2074042288752348E15,1.10951746294648096E17]|
|10   |[1.6131199347666342E19,1.24546214832341616E17,8.5265750194040304E16] |
|2    |[4.330324858747168E12,6.19671483053885E12,2.2416578004282832E13]     |
+-----+---------------------------------------------------------------------+

注意


  • 请注意, MultivariateOnlineSummarizer 需要旧样式 mllib.linalg.Vector ml.linalg.Vector 不能使用。为了支持这些,您必须在新旧类型之间转换

  • 在性能方面,您可能会使用 RDD 更好。 / li>
  • Please note that MultivariateOnlineSummarizer requires "old style" mllib.linalg.Vector. It won't work with ml.linalg.Vector. To support these you'll have to convert between new and old types.
  • Performance wise you'll probably be better off with RDDs.

这篇关于如何在Spark SQL中找到分组的Vector列的均值?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆