动态框架解析选择规格，日期转换 [英] Dynamic frame resolve choice specs , date cast

问题描述

我正在编写Glue代码，并使用动态框架Api解决选择规范，即specs.我正在尝试通过从目录创建动态框架时传递强制转换来强制转换源. 我已经通过解析选择规范成功实现了转换，但是在转换日期时我得到了空值，只是想了解如何在转换中使用源格式传递日期. self.df_TR01=self.df_TR01.resolveChoice(specs=[('col1', 'cast"string'), ('col2_date', 'cast:date')]).toDF()

I am writing a Glue code and using dynamic frame Api resolve choice , specs . I am trying to cast the source by passing casting when dynamic frame is created from catalog. I have successfully implemented the casting via resolve choice specs but while casting date i am getting null values , just wanted to understand how can we pass date with source format in casting. self.df_TR01=self.df_TR01.resolveChoice(specs=[('col1', 'cast"string'), ('col2_date', 'cast:date')]).toDF()

但是在col2_date中，我得到的是空值，并且我试图了解如何在上述语句中以源格式传递日期.

But in col2_date i am getting null value and i am trying to understand how can i pass date with source format in the above statement.

推荐答案

我遇到了类似的问题，但是在向Redshift写入日期之前，我的问题是它们也为空值.就我而言，我使用了以下内容，它帮助我解决了该问题.也许这会有所帮助.

I encountered something similar but my issue before when writing dates to Redshift, they were also landing as nulls. In my case, I used the following and it helped me resolve the issue. Maybe this will help.

from datetime import datetime

def fix_dates(m):
    m["col2"] = datetime.strptime(m["col2"],  "m/d/yy")
    return m

custommapping1 = Map.apply(frame = datasource0, f = fix_dates, transformation_ctx = "custommapping1")

或者，您可以使用spark sql，例如:

Alternatively you can make use of spark sql, for example:

datasource0.toDF().createOrReplaceTempView("my_temp_view")

df_cols  = spark.sql("""
  select to_date(cast(unix_timestamp(col2, 'M/d/yy') as timestamp)) as col2 from my_temp_view """)

ResolveChoice通常能够处理大多数歧义.您能否分享无法正确投放的样本日期，也许我也可以尝试一下.

ResolveChoice is normally able to deal with most ambiguities. Can you share a sample date that is failing to cast correctly, maybe I could also try on my end.

这篇关于动态框架解析选择规格，日期转换的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！