具有高阶功能的火花过滤器 [英] spark filter with higher order function
本文介绍了具有高阶功能的火花过滤器的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
如何在Scala中获得一个高阶函数以正确接受火花过滤器谓词?IE.
How can I get a higher order function in Scala to properly accept a spark filter predicate? I.e.
val df = Seq(1,2,3,4).toDF("value")
df.filter(col("value")> 2).show
df.filter(col("value")< 2).show
工作正常.但是,当我尝试将其重构为接受过滤谓词(请注意:与>
运算符相同的签名)的函数时,编译器将不再找到要提交给谓词的左/右部分.
works just fine. But when I try to refactor it into a function which accepts a filter predicate (note: same signature as >
operator) the compiler no longer finds the left/right part to submit to predicate.
def myFilter =(predicate:Any =>Column)(df:DataFrame)= {
df.filter(col("value") predicate 2).show // WARN this does not compile
}
df.transform(myFilter(>)).show
如何使它起作用?
推荐答案
将各种注释组合起来,可以作为一种可能的解决方案:
Combining the various comments gives this as a possible solution:
def myFilter = (predicate: (Column, Int) => Column)(df: DataFrame) = {
df.filter(predicate(col("value"), 2))
}
df.transform(myFilter(_ > _)).show
这篇关于具有高阶功能的火花过滤器的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文