从Spark 2.0到3.0的从字符串到日期的迁移使DateTimeFormatter中无法识别'EEE MMM dd HH:mm:ss zzz yyyy'模式 [英] String to Date migration from Spark 2.0 to 3.0 gives Fail to recognize 'EEE MMM dd HH:mm:ss zzz yyyy' pattern in the DateTimeFormatter

问题描述

我有一个来源字符串中的日期字符串，格式为"Fri May 24 00:00:00 BST 2019"，我将使用我的代码将其转换为日期并以"2019-05-24"的形式存储在我的数据框中在spark 2.0下对我有效的示例

I have a date string from a source in the format 'Fri May 24 00:00:00 BST 2019' that I would convert to a date and store in my dataframe as '2019-05-24' using code like my example which works for me under spark 2.0

from pyspark.sql.functions import to_date, unix_timestamp, from_unixtime
df = spark.createDataFrame([("Fri May 24 00:00:00 BST 2019",)], ['date_str'])
df2 = df.select('date_str', to_date(from_unixtime(unix_timestamp('date_str', 'EEE MMM dd HH:mm:ss zzz yyyy'))).alias('date'))
df2.show(1, False)

在我的沙盒环境中，我已经更新为spark 3.0，现在上面的代码收到以下错误，是否有一种新的方法可以在3.0中执行以将我的字符串转换为日期

In my sandbox environment I've updated to spark 3.0 and now get the following error for the above code, is there a new method of doing this in 3.0 to convert my string to a date

:org.apache.spark.SparkUpgradeException:您可能会得到一个不同的结果Spark 3.0升级导致的结果:无法识别'EEE MMM日期时间格式设置中的dd HH:mm:ss zzz yyyy'模式.

: org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to recognize 'EEE MMM dd HH:mm:ss zzz yyyy' pattern in the DateTimeFormatter.

1. 您可以将spark.sql.legacy.timeParserPolicy设置为LEGACY以恢复Spark 3.0之前的行为.
2. 您可以使用以下指南创建有效的日期时间格式: https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html

推荐答案

如果要在更新版本的spark(> 3)中使用旧格式，则需要设置 spark.conf.set("spark.sql.legacy.timeParserPolicy"，"LEGACY")或 spark.sql("set spark.sql.legacy.timeParserPolicy = LEGACY")，它将解决此问题.

If you want to use the legacy format in a newer version of spark(>3), you need to set spark.conf.set("spark.sql.legacy.timeParserPolicy","LEGACY") or spark.sql("set spark.sql.legacy.timeParserPolicy=LEGACY"), which will resolve the issue.

这篇关于从Spark 2.0到3.0的从字符串到日期的迁移使DateTimeFormatter中无法识别'EEE MMM dd HH:mm:ss zzz yyyy'模式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！