Spark - 选择 WHERE 还是过滤? [英] Spark - SELECT WHERE or filtering?

问题描述

在 Spark 中使用 where 子句进行选择和过滤有什么区别?
是否有任何用例比另一种更合适?

什么时候用

DataFrame newdf = df.select(df.col("*")).where(df.col("somecol").leq(10))

什么时候

DataFrame newdf = df.select(df.col("*")).filter("somecol <= 10")

更合适?

解决方案

根据 spark 文档 "where() 是 filter() 的别名"

过滤器(条件)使用给定条件过滤行.where() 是 filter() 的别名.

参数:条件 - types.BooleanType 的 Column 或 SQL 表达式字符串.

<预><代码>>>>df.filter(df.age > 3).collect()[Row(age=5, name=u'Bob')]>>>df.where(df.age == 2).collect()[Row(age=2, name=u'Alice')]>>>df.filter("age > 3").collect()[Row(age=5, name=u'Bob')]>>>df.where("age = 2").collect()[Row(age=2, name=u'Alice')]

What's the difference between selecting with a where clause and filtering in Spark?
Are there any use cases in which one is more appropriate than the other one?

When do I use

DataFrame newdf = df.select(df.col("*")).where(df.col("somecol").leq(10))

and when is

DataFrame newdf = df.select(df.col("*")).filter("somecol <= 10")

more appropriate?

解决方案

According to spark documentation "where() is an alias for filter()"

filter(condition) Filters rows using the given condition. where() is an alias for filter().

Parameters: condition – a Column of types.BooleanType or a string of SQL expression.

>>> df.filter(df.age > 3).collect()
[Row(age=5, name=u'Bob')]
>>> df.where(df.age == 2).collect()
[Row(age=2, name=u'Alice')]

>>> df.filter("age > 3").collect()
[Row(age=5, name=u'Bob')]
>>> df.where("age = 2").collect()
[Row(age=2, name=u'Alice')]

这篇关于Spark - 选择 WHERE 还是过滤?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！