Python API 中是否提供 Spark SQL UDAF(用户定义的聚合函数)? [英] Is Spark SQL UDAF (user defined aggregate function) available in the Python API?

问题描述

从 Spark 1.5.0 开始，似乎可以为 DataFrame 上的自定义聚合编写自己的 UDAF:Spark 1.5DataFrame API 亮点:日期/时间/字符串处理、时间间隔和 UDAF

As of Spark 1.5.0 it seems possible to write your own UDAF's for custom aggregations on DataFrames: Spark 1.5 DataFrame API Highlights: Date/Time/String Handling, Time Intervals, and UDAFs

但我不清楚 Python API 是否支持此功能?

It is however unclear to me if this functionality is supported in the Python API?

推荐答案

您无法在 Spark 1.5.0-2.0.0 中定义 Python UDAF.有一个 JIRA 跟踪此功能请求:

You cannot defined Python UDAF in Spark 1.5.0-2.0.0. There is a JIRA tracking this feature request:

• https://issues.apache.org/jira/browse/SPARK-10915

稍后"解决了目标，因此它可能不会很快发生.

resolved with goal "later" so it probably won't happen anytime soon.

您可以使用 PySpark 中的 Scala UDAF - 描述为 Spark:如何使用 Scala 或 Java 用户定义函数映射 Python?

You can use Scala UDAF from PySpark - it is described Spark: How to map Python with Scala or Java User Defined Functions?

这篇关于Python API 中是否提供 Spark SQL UDAF(用户定义的聚合函数)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！