咖喱 UDF - Pyspark [英] Curried UDF - Pyspark
问题描述
我正在尝试在 spark 中实现一个 UDF;可以将文字和列作为参数.为了实现这一点,我相信我可以使用咖喱 UDF.
I am trying to implement a UDF in spark; that can take both a literal and column as an argument. To achieve this, I believe I can use a curried UDF.
该函数用于将字符串文字与 DataFrame
列中的每个值进行匹配.我总结了以下代码:-
The function is used to match a string literal to each value in the column of a DataFrame
. I have summarized the code below:-
def matching(match_string_1):
def matching_inner(match_string_2):
return difflib.SequenceMatcher(None, match_string_1, match_string_2).ratio()
return matching
hc.udf.register("matching", matching)
matching_udf = F.udf(matching, StringType())
df_matched = df.withColumn("matching_score", matching_udf(lit("match_string"))(df.column))
"match_string"
实际上是分配给我正在迭代的列表的值."match_string"
is actually a value assigned to a list which I am iterating over.
不幸的是,这并没有像我希望的那样工作;我收到了
Unfortunately this is not working as I had hoped; and I am receiving
类型错误:‘列’对象不可调用".
"TypeError: 'Column' object is not callable".
我相信我没有正确调用这个函数.
I believe I am not calling this function correctly.
推荐答案
应该是这样的:
def matching(match_string_1):
def matching_inner(match_string_2):
return difflib.SequenceMatcher(
a=match_string_1, b=match_string_2).ratio()
# Here create udf.
return F.udf(matching_inner, StringType())
df.withColumn("matching_score", matching("match_string")(df.column))
如果你想支持 match_string_1
的 Column
参数,你必须像这样重写它:
If you want to support Column
argument for match_string_1
you'll have to rewrite it like this:
def matching(match_string_1):
def matching_inner(match_string_2):
return F.udf(
lambda a, b: difflib.SequenceMatcher(a=a, b=b).ratio(),
StringType())(match_string_1, match_string_2)
return matching_inner
df.withColumn("matching_score", matching(F.lit("match_string"))(df.column)
您当前的代码不起作用,matching_udf
是 UDF 和 matching_udf(lit("match_string"))
创建一个 Column
表达式而不是调用内部函数.
Your current code doesn't work, matching_udf
is and UDF and matching_udf(lit("match_string"))
creates a Column
expression instead of calling internal function.
这篇关于咖喱 UDF - Pyspark的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!