Spark SQL过滤多个字段 [英] Spark SQL filter multiple fields

问题描述

在Scala API中对多列进行过滤的相关语法是什么?如果我想做这样的事情:

What is the corrent syntax for filtering on multiple columns in the Scala API? If I want to do something like this:

dataFrame.filter($"col01" === "something" && $"col02" === "something else")

或

dataFrame.filter($"col01" === "something" || $"col02" === "something else") 

这是我原始代码的样子.一切都以字符串形式出现.

This is what my original code looks like. Everything comes in as a string.

df.select($"userID" as "user", $"itemID" as "item", $"quantity" cast("int"), $"price" cast("float"), $"discount" cast ("float"), sqlf.substring($"datetime", 0, 10) as "date", $"group")
  .filter($"item" !== "" && $"group" !== "-1")

推荐答案

我想我明白了问题所在.由于某种原因，spark不允许在同一 filter 中使用两个！=.需要查看如何在Spark源代码中定义 过滤器 .

I think i see what the issue is. For some reason, spark does not allow two !='s in the same filter. Need to look at how filter is defined in Spark source code.

现在您的代码可以正常工作了，您可以使用它进行过滤

Now for your code to work, you can use this to do the filter

df.filter(col("item").notEqual("") && col("group").notEqual("-1"))

或在同一条语句中使用两个过滤器

or use two filters in same statement

df.filter($"item" !== "").filter($"group" !== "-1").select(....)

此链接此处可以帮助使用不同的火花方法.

This link here can help with different spark methods.

这篇关于Spark SQL过滤多个字段的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！