Spark SQL datediff以秒为单位 [英] Spark SQL datediff in seconds

问题描述

我有以下代码:

table.select(datediff(table.col("Start Time"), table.col("End Time"))).show()

日期格式为 2016-05-19 09:23:28(YYYY-MM-DD HH:mm:SS)

函数 datediff 计算天数差异.但我想在几秒钟内有所不同.

Function datediff calculate the difference in days. But I would like to have the difference in seconds.

推荐答案

您可以使用

You can use unix_timestamp() function to convert date to seconds.

import org.apache.spark.sql.functions._

//For $ notation columns // Spark 2.0
import spark.implicits._

table.withColumn("date_diff", 
   (unix_timestamp($"Start Time") - unix_timestamp($"End Time"))
).show()

(根据评论)

sqlContext.udf.register("sec_to_time", (s: Long) => 
   ((s / 3600L) + ":" + (s / 60L) + ":" + (s % 60L))
)

//Use registered UDF now
table.withColumn("date_diff", 
   sec_to_time(unix_timestamp($"Start Time") - unix_timestamp($"End Time"))
).show()

这篇关于Spark SQL datediff以秒为单位的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！