如何获取每个条目的所有行条目中scala-spark中数组类型列的平均值? [英] How to obtain the average of an array-type column in scala-spark over all row entries per entry?
本文介绍了如何获取每个条目的所有行条目中scala-spark中数组类型列的平均值?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我有一个包含512个双元素的数组列,想要获得平均值.以length = 3的数组列为例:
I got an array column with 512 double elements, and want to get the average. Take an array column with length=3 as example:
val x = Seq("2 4 6", "0 0 0").toDF("value").withColumn("value", split($"value", " "))
x.printSchema()
x.show()
root
|-- value: array (nullable = true)
| |-- element: string (containsNull = true)
+---------+
| value|
+---------+
|[2, 4, 6]|
|[0, 0, 0]|
+---------+
需要以下结果:
x.select(..... as "avg_value").show()
------------
|avg_value |
------------
|[1,2,3] |
------------
推荐答案
将每个数组元素视为列并计算平均值,然后使用这些列构造数组:
Consider each array element as column and calculate average then construct array with those columns:
val array_size = 3
val avgAgg = for (i <- 0 to array_size -1) yield avg($"value".getItem(i))
df.select(array(avgAgg: _*).alias("avg_value")).show(false)
赠予:
+---------------+
|avg_value |
+---------------+
|[1.0, 2.0, 3.0]|
+---------------+
这篇关于如何获取每个条目的所有行条目中scala-spark中数组类型列的平均值?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文