Spark Scala:按小时或分钟显示两列的DateDiff [英] Spark Scala: DateDiff of two columns by hour or minute
本文介绍了Spark Scala:按小时或分钟显示两列的DateDiff的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我在数据框中有两个时间戳列,我想获得它们的分钟差异,或者小时差异.目前,我可以通过四舍五入来获得日差
I have two timestamp columns in a dataframe that I'd like to get the minute difference of, or alternatively, the hour difference of. Currently I'm able to get the day difference, with rounding, by doing
val df2 = df1.withColumn("time", datediff(df1("ts1"), df1("ts2")))
但是,当我查看文档页面时https://issues.apache.org/jira/browse/SPARK-8185我没有看到任何额外的参数来改变单位.我应该为此使用它们的不同功能吗?
However, when i looked at the doc page https://issues.apache.org/jira/browse/SPARK-8185 I didn't see any extra parameters to change the unit. Is their a different function I should be using for this?
推荐答案
你可以通过
import org.apache.spark.sql.functions._
val diff_secs_col = col("ts1").cast("long") - col("ts2").cast("long")
然后你可以做一些数学运算来得到你想要的单位.例如:
Then you can do some math to get the unit you want. For example:
val df2 = df1
.withColumn( "diff_secs", diff_secs_col )
.withColumn( "diff_mins", diff_secs_col / 60D )
.withColumn( "diff_hrs", diff_secs_col / 3600D )
.withColumn( "diff_days", diff_secs_col / (24D * 3600D) )
或者,在 pyspark 中:
Or, in pyspark:
from pyspark.sql.functions import *
diff_secs_col = col("ts1").cast("long") - col("ts2").cast("long")
df2 = df1 \
.withColumn( "diff_secs", diff_secs_col ) \
.withColumn( "diff_mins", diff_secs_col / 60D ) \
.withColumn( "diff_hrs", diff_secs_col / 3600D ) \
.withColumn( "diff_days", diff_secs_col / (24D * 3600D) )
这篇关于Spark Scala:按小时或分钟显示两列的DateDiff的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文