链接Dataframe函数调用 [英] Chaining Dataframe function calls

问题描述

以下代码不起作用:

val newDF = df
          .withColumn("timestamp", when(df("processingDate").isNull, lit(new Timestamp(System.currentTimeMillis))).otherwise(df("processingDate")))
          .withColumn("year", year(df("timestamp")))
          .withColumn("month", month(df("timestamp")))
          .withColumn("day", dayofmonth(df("timestamp")))

如果我运行它，将会得到以下异常:

If I run it I will get the following exception:

Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot resolve column name "timestamp" among ...

问题是，尽管我已将时间戳记"添加为列，但它不是原始的，不可变的"df"的一部分.

The problem is that although I have added "timestamp" as a column, it is not part of the original, immutable, "df".

有没有办法引用调用链中的上一个数据框?

Is there a way to refer to the previous Dataframe in the call chain?

我将代码更新为以下代码，以便可以使用，但是我想知道是否有更好的方法.

I will update my code to the following so that it will work, but I am wondering if there is a better way.

val dfWithTimestamp = df.withColumn("timestamp", when(df("monBusinessDateTimestamp").isNull, lit(new Timestamp(System.currentTimeMillis))).otherwise(df("monBusinessDateTimestamp")))

val newDF = dfWithTimestamp
          .withColumn("year", year(dfWithTimestamp("timestamp")))
          .withColumn("month", month(dfWithTimestamp("timestamp")))
          .withColumn("day", dayofmonth(dfWithTimestamp("timestamp")))

推荐答案

我现在无法检查，但是

val newDF = df
          .withColumn("timestamp", when(df("processingDate").isNull, lit(new Timestamp(System.currentTimeMillis))).otherwise(df("processingDate")))
          .withColumn("year", year($"timestamp"))
          .withColumn("month", month($"timestamp"))
          .withColumn("day", dayofmonth($"timestamp"))

可能会工作.

这篇关于链接Dataframe函数调用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！