具有高阶函数的火花过滤器 [英] spark filter with higher order function

问题描述

如何在 Scala 中获得高阶函数以正确接受火花过滤器谓词?IE.

How can I get a higher order function in Scala to properly accept a spark filter predicate? I.e.

val df = Seq(1,2,3,4).toDF("value")

df.filter(col("value")> 2).show
df.filter(col("value")< 2).show

工作正常.但是当我尝试将它重构为一个接受过滤谓词的函数时(注意:与 > 运算符相同的签名)编译器不再找到要提交给谓词的左/右部分.

works just fine. But when I try to refactor it into a function which accepts a filter predicate (note: same signature as > operator) the compiler no longer finds the left/right part to submit to predicate.

def myFilter =(predicate:Any =>Column)(df:DataFrame)= {
df.filter(col("value") predicate 2).show // WARN this does not compile
}

df.transform(myFilter(>)).show

如何才能做到这一点?

推荐答案

结合各种评论给出了一个可能的解决方案:

Combining the various comments gives this as a possible solution:

def myFilter = (predicate: (Column, Int) => Column)(df: DataFrame) = {
  df.filter(predicate(col("value"), 2))
}

df.transform(myFilter(_ > _)).show

这篇关于具有高阶函数的火花过滤器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！