星火斯卡拉：两列则DateDiff按小时或分钟 [英] Spark Scala: DateDiff of two columns by hour or minute

问题描述

我在，我想获得的分差，或者，对小时的时差一个数据帧2时间戳列。目前，我能够得到天差异，舍入，做

I have two timestamp columns in a dataframe that I'd like to get the minute difference of, or alternatively, the hour difference of. Currently I'm able to get the day difference, with rounding, by doing

val df2 = df1.withColumn("time", datediff(df1("ts1"), df1("ts2")))

然而，当我看着文档页面
https://issues.apache.org/jira/browse/SPARK-8185
我没有看到任何额外的参数来改变单元。是他们，我应该使用这种不同的功能？

However, when i looked at the doc page https://issues.apache.org/jira/browse/SPARK-8185 I didn't see any extra parameters to change the unit. Is their a different function I should be using for this?

推荐答案

您可以通过

import org.apache.spark.sql.functions._
val diff_secs_col = col("ts1").cast("long") - col("ts2").cast("long")

然后，你可以做一些数学得到你想要的单位。例如：

Then you can do some math to get the unit you want. For example:

val df2 = df1
  .withColumn( "diff_secs", diff_secs_col )
  .withColumn( "diff_mins", diff_secs_col / 60D )
  .withColumn( "diff_hrs",  diff_secs_col / 3600D )
  .withColumn( "diff_days", diff_secs_col / (24D * 3600D) )

或者，在pyspark：

Or, in pyspark:

from pyspark.sql.functions import *
diff_secs_col = col("ts1").cast("long") - col("ts2").cast("long")

df2 = df1 \
  .withColumn( "diff_secs", diff_secs_col ) \
  .withColumn( "diff_mins", diff_secs_col / 60D ) \
  .withColumn( "diff_hrs",  diff_secs_col / 3600D ) \
  .withColumn( "diff_days", diff_secs_col / (24D * 3600D) )

这篇关于星火斯卡拉：两列则DateDiff按小时或分钟的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！