to_date 无法解析 Spark 3.0 中的日期 [英] to_date fails to parse date in Spark 3.0

问题描述

我正在尝试使用 to_date() 解析日期，但出现以下异常.

I am trying to parse date using to_date() but I get the following exception.

SparkUpgradeException:由于 Spark 3.0 升级，您可能会得到不同的结果:在新的解析器中无法解析 '12/1/2010 8:26'.您可以将 spark.sql.legacy.timeParserPolicy 设置为 LEGACY 以恢复 Spark 3.0 之前的行为，或者设置为 CORRECTED 并将其视为无效的日期时间字符串.

SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to parse '12/1/2010 8:26' in the new parser. You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0, or set to CORRECTED and treat it as an invalid datetime string.

例外情况表明我应该使用旧版时间解析器，首先我不知道如何将其设置为旧版.

The exception suggests I should use a legacy Time Parser, for starter I don't know how to set it to Legacy.

这是我的实现

dfWithDate = df.withColumn("date", to_date(col("InvoiceDate"), "MM/dd/yyyy"))

我的日期格式如下

+--------------+
|   InvoiceDate|
+--------------+
|12/1/2010 8:26|
|12/1/2010 8:26|
|12/1/2010 8:26|
|12/1/2010 8:26|
|12/1/2010 8:26|

推荐答案

spark.sql("set spark.sql.legacy.timeParserPolicy=LEGACY")
df.withColumn("date", to_date(col("InvoiceDate"), "MM/dd/yyyy")).show()


+--------------+----------+
|   InvoiceDate|      date|
+--------------+----------+
|12/1/2010 8:26|2010-12-01|
+--------------+----------+

# in above code spark refers SparkSession

这篇关于to_date 无法解析 Spark 3.0 中的日期的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！