UDF的Pyspark错误：py4j.Py4JException：方法__getnewargs __（[]）不存在错误 [英] Pyspark error with UDF: py4j.Py4JException: Method __getnewargs__([]) does not exist error

问题描述

我正在尝试解决以下错误（我正在使用databricks平台和spark 2.0）

I am trying to solve the following error (I am using the databricks platform and spark 2.0)

tweets_cleaned.createOrReplaceTempView("tweets_cleanedSQL")
def Occ(keyword):
  occurences = spark.sql("SELECT * \
                                FROM tweets_cleanedSQL \
                                WHERE LOWER(text) LIKE '%" + keyword + "%' \
                            ")
  return occurences.count()


occurences_udf = udf(Occ)

如果运行此代码，则会收到以下错误：

If I run this code, I receive the following error:

py4j。 Py4JException：方法 getnewargs （[]）不存在==>仅在尝试定义udf时发生错误。

py4j.Py4JException: Method getnewargs([]) does not exist ==> error only occurs when trying to define the udf.

推荐答案

UDF函数将常规函数转换为应用于输入列的任何元素的函数。您不能使用此函数调用spark函数（在这种情况下，您正在调用spark.sql，这反过来又需要创建worker等）。

the UDF function turns a regular function to a function that is applied on any element of the input column. You cannot have this function call spark functions (in this case you are calling spark.sql which would in turn need to create workers etc. which is not supported.

这篇关于UDF的Pyspark错误：py4j.Py4JException：方法__getnewargs __（[]）不存在错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！