在 PySpark 中使用时区在日期时间范围之间过滤镶木地板文件 [英] Filter between datetime ranges with timezone in PySpark for parquet files
问题描述
基于 此处,我想知道如何使用 PySpark 过滤带有时区的日期时间范围.
Based on the suggestion from here, I would like to know how do I filter the datetime ranges with timezone using PySpark.
我的数据如下所示:
ABC, 2020-06-22T19:17:16.428+0000
ABC, 2020-06-22T19:17:16.428+0000
DEF,2020-06-22T19:17:16.435+0000
DEF, 2020-06-22T19:17:16.435+0000
JKL,2020-06-22T19:17:16.468+0000
JKL, 2020-06-22T19:17:16.468+0000
移动网络运营商,2020-06-22T19:17:16.480+0000
MNO, 2020-06-22T19:17:16.480+0000
XYZ, 2020-06-22T19:17:16.495+0000
XYZ, 2020-06-22T19:17:16.495+0000
在这种情况下,我只想提取那些毫秒数在 400-450 之间的记录.
I would only like to extract those records that has milliseconds between 400-450 in this case.
试过了,但没有用:
import pyspark.sql.functions as func
df = df.select(func.to_date(df.UpdatedOn).alias("time"))
sf = df.filter(df.time > '2020-06-22T19:17:16.400').filter(df.time < '2020-06-22T19:17:16.451')
推荐答案
当你使用 to_date
时它会截断时间,所以你必须使用 to_timestamp
并比较它.
When you use to_date
it will truncate the hours, so you have to use to_timestamp
and compare it.
df.withColumn('date', to_timestamp('date')) \
.filter("date between to_timestamp('2020-06-22T19:17:16.400') and to_timestamp('2020-06-22T19:17:16.451')") \
.show(10, False)
+---+-----------------------+
|id |date |
+---+-----------------------+
|ABC|2020-06-22 19:17:16.428|
|DEF|2020-06-22 19:17:16.435|
+---+-----------------------+
这篇关于在 PySpark 中使用时区在日期时间范围之间过滤镶木地板文件的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!