带时间的火花请求 [英] Spark Request with time

问题描述

我想请求平均以1小时为间隔查找一天中最忙的时间.

I want to make a request to find the most busy time of the day on average in 1-hour intervals.

我在数据框中有行日期，格式为％d/％b/％Y:％H:％M:％S".

I have on my dataframe the row date in format "%d/%b/%Y:%H:%M:%S".

我是这样开始的:

mostBusyTimeDF = logDF.groupBy("date") ...

例如输入:

               date
 2015-12-01 21:04:00
 2015-12-01 10:04:00
 2015-12-01 21:07:00
 2015-12-01 21:34:00

在输出中:

               date         count(1 hour interval)
 2015-12-01 21:04:00                          3
 2015-12-01 10:04:00                          1

在我不知道该怎么办之后..

After I don't know how can I do it..

你能帮我吗?

非常感谢

推荐答案

您可以使用内置的Spark日期函数:

You can use built-in Spark date functions:

from pyspark.sql.functions import *

logDF = sqlContext.createDataFrame([("2015-12-01 21:04:00", 1), ("2015-12-01 10:04:00", 2), ("2015-12-01 21:07:00", 9), ("2015-12-01 21:34:00", 1)], ['somedate', 'someother'])

busyTimeDF = logDF.groupBy(year("somedate").alias("cnt_year"), \
    month("somedate").alias("cnt_month"), \
    dayofmonth("somedate").alias("cnt_day"), \
    hour('somedate').alias("cnt_hour")) \
       .agg(functions.count("*").alias("cntHour")) 

cond = [busyTimeDF.cnt_year == year(logDF.somedate), \
    busyTimeDF.cnt_month == month(logDF.somedate), \
    busyTimeDF.cnt_day == dayofmonth(logDF.somedate), \
    busyTimeDF.cnt_hour == hour(logDF.somedate)]

busyTimeDF.join(logDF, cond).select('somedate', 'cntHour').show()

这篇关于带时间的火花请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！