当值与pyspark中的字符串的一部分匹配时过滤df [英] Filter df when values matches part of a string in pyspark

问题描述

我有一个很大的pyspark.sql.dataframe.DataFrame，我想保留(所以filter)所有保存在location列中的URL包含预定字符串的行，例如"google.com".

I have a large pyspark.sql.dataframe.DataFrame and I want to keep (so filter) all rows where the URL saved in the location column contains a pre-determined string, e.g. 'google.com'.

我尝试过:

import pyspark.sql.functions as sf
df.filter(sf.col('location').contains('google.com')).show(5)

但这会引发

TypeError: _TypeError: 'Column' object is not callable'

如何解决并正确过滤我的df?提前谢谢了！

How do I go around and filter my df properly? Many thanks in advance!

推荐答案

Spark 2.2及更高版本

df.filter(df.location.contains('google.com'))

Spark 2.2文档链接

---

Spark 2.1及更低版本

您可以在filter

df.filter("location like '%google.com%'")

或使用DataFrame列方法

df.filter(df.location.like('%google.com%'))

查看全文