数据框上的多条件过滤器 [英] Multiple condition filter on dataframe

问题描述

任何人都可以向我解释为什么我对这两个表达式会得到不同的结果吗?我正在尝试在2个日期之间进行过滤:

Can anyone explain to me why I am getting different results for these 2 expressions ? I am trying to filter between 2 dates:

df.filter("act_date <='2017-04-01'" and "act_date >='2016-10-01'")\
  .select("col1","col2").distinct().count()

结果:3700万

vs

df.filter("act_date <='2017-04-01'").filter("act_date >='2016-10-01'")\
  .select("col1","col2").distinct().count()

结果:2500万

它们有何不同?在我看来，他们应该产生相同的结果

How are they different ? It seems to me like they should produce the same result

推荐答案

TL; DR 要将多个条件传递给filter或where，请使用Column对象和逻辑运算符(，|，~).参见 Pyspark:when子句中的多个条件.

TL;DR To pass multiple conditions to filter or where use Column objects and logical operators (&, |, ~). See Pyspark: multiple conditions in when clause.

df.filter((col("act_date") >= "2016-10-01") & (col("act_date") <= "2017-04-01"))

您还可以使用单个 SQL字符串:

You can also use a single SQL string:

df.filter("act_date >='2016-10-01' AND act_date <='2017-04-01'")

在实践中，在以下两者之间使用更有意义:

In practice it makes more sense to use between:

df.filter(col("act_date").between("2016-10-01", "2017-04-01"))
df.filter("act_date BETWEEN '2016-10-01' AND '2017-04-01'")

第一种方法甚至不是远程有效的.在Python中，and返回:

The first approach is not even remote valid. In Python, and returns:

• 如果所有表达式均为"truthy"，则为最后一个元素.
• 否则，第一个假"元素.

结果

"act_date <='2017-04-01'" and "act_date >='2016-10-01'"

评估为(任何非空字符串为真):

is evaluated to (any non-empty string is truthy):

"act_date >='2016-10-01'"

这篇关于数据框上的多条件过滤器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！