在PySpark中使用时区在日期时间范围之间筛选实木复合地板文件 [英] Filter between datetime ranges with timezone in PySpark for parquet files

问题描述

基于

Based on the suggestion from here, I would like to know how do I filter the datetime ranges with timezone using PySpark.

这是我的数据的样子:

美国广播公司(ABC)，2020-06-22T19:17:16.428 + 0000

ABC, 2020-06-22T19:17:16.428+0000

DEF，2020-06-22T19:17:16.435 + 0000

DEF, 2020-06-22T19:17:16.435+0000

JKL，2020-06-22T19:17:16.468 + 0000

JKL, 2020-06-22T19:17:16.468+0000

移动网络运营商，2020-06-22T19:17:16.480 + 0000

MNO, 2020-06-22T19:17:16.480+0000

XYZ，2020-06-22T19:17:16.495 + 0000

XYZ, 2020-06-22T19:17:16.495+0000

在这种情况下，我只想提取毫秒数在400-450之间的记录.

I would only like to extract those records that has milliseconds between 400-450 in this case.

尝试了一下，但是没有用:

Tried this but didn't work:

import pyspark.sql.functions as func
df = df.select(func.to_date(df.UpdatedOn).alias("time"))
sf = df.filter(df.time > '2020-06-22T19:17:16.400').filter(df.time < '2020-06-22T19:17:16.451')

推荐答案

使用to_date会截断小时数，因此必须使用to_timestamp进行比较.

When you use to_date it will truncate the hours, so you have to use to_timestamp and compare it.

df.withColumn('date', to_timestamp('date')) \
  .filter("date between to_timestamp('2020-06-22T19:17:16.400') and to_timestamp('2020-06-22T19:17:16.451')") \
  .show(10, False)

+---+-----------------------+
|id |date                   |
+---+-----------------------+
|ABC|2020-06-22 19:17:16.428|
|DEF|2020-06-22 19:17:16.435|
+---+-----------------------+

这篇关于在PySpark中使用时区在日期时间范围之间筛选实木复合地板文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！