链接数据帧函数调用 [英] Chaining Dataframe function calls

问题描述

以下代码不起作用:

val newDF = df
          .withColumn("timestamp", when(df("processingDate").isNull, lit(new Timestamp(System.currentTimeMillis))).otherwise(df("processingDate")))
          .withColumn("year", year(df("timestamp")))
          .withColumn("month", month(df("timestamp")))
          .withColumn("day", dayofmonth(df("timestamp")))

如果我运行它，我会得到以下异常:

If I run it I will get the following exception:

Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot resolve column name "timestamp" among ...

问题是，虽然我添加了时间戳"作为一列，但它不是原始的、不可变的df"的一部分.

The problem is that although I have added "timestamp" as a column, it is not part of the original, immutable, "df".

有没有办法在调用链中引用前一个Dataframe?

Is there a way to refer to the previous Dataframe in the call chain?

我会将我的代码更新为以下内容，以便它可以工作，但我想知道是否有更好的方法.

I will update my code to the following so that it will work, but I am wondering if there is a better way.

val dfWithTimestamp = df.withColumn("timestamp", when(df("monBusinessDateTimestamp").isNull, lit(new Timestamp(System.currentTimeMillis))).otherwise(df("monBusinessDateTimestamp")))

val newDF = dfWithTimestamp
          .withColumn("year", year(dfWithTimestamp("timestamp")))
          .withColumn("month", month(dfWithTimestamp("timestamp")))
          .withColumn("day", dayofmonth(dfWithTimestamp("timestamp")))

推荐答案

我现在无法检查，但是

val newDF = df
          .withColumn("timestamp", when(df("processingDate").isNull, lit(new Timestamp(System.currentTimeMillis))).otherwise(df("processingDate")))
          .withColumn("year", year($"timestamp"))
          .withColumn("month", month($"timestamp"))
          .withColumn("day", dayofmonth($"timestamp"))

可能有用.

这篇关于链接数据帧函数调用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！