PySpark 数字窗口分组依据 [英] PySpark Numeric Window Group By
本文介绍了PySpark 数字窗口分组依据的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我希望能够按步长设置 Spark 组,而不仅仅是单个值.spark 中是否有类似于 PySpark 2.x 的 window
函数用于数字(非日期)值?
I'd like to be able to have Spark group by a step size, as opposed to just single values. Is there anything in spark similar to PySpark 2.x's window
function for numeric (non-date) values?
大致如下:
sqlContext = SQLContext(sc)
df = sqlContext.createDataFrame([10, 11, 12, 13], "integer").toDF("foo")
res = df.groupBy(window("foo", step=2, start=10)).count()
推荐答案
您可以重复使用时间戳一个并以秒为单位表达参数.翻滚:
You can reuse timestamp one and express parameters in seconds. Tumbling:
from pyspark.sql.functions import col, window
df.withColumn(
"window",
window(
col("foo").cast("timestamp"),
windowDuration="2 seconds"
).cast("struct<start:bigint,end:bigint>")
).show()
# +---+-------+
# |foo| window|
# +---+-------+
# | 10|[10,12]|
# | 11|[10,12]|
# | 12|[12,14]|
# | 13|[12,14]|
# +---+-------+
滚动一:
df.withColumn(
"window",
window(
col("foo").cast("timestamp"),
windowDuration="2 seconds", slideDuration="1 seconds"
).cast("struct<start:bigint,end:bigint>")
).show()
# +---+-------+
# |foo| window|
# +---+-------+
# | 10| [9,11]|
# | 10|[10,12]|
# | 11|[10,12]|
# | 11|[11,13]|
# | 12|[11,13]|
# | 12|[12,14]|
# | 13|[12,14]|
# | 13|[13,15]|
# +---+-------+
使用 groupBy
和 start
:
w = window(col("foo").cast("timestamp"), "2 seconds").cast("struct<start:bigint,end:bigint>")
start = w.start.alias("start")
df.groupBy(start).count().show()
+-----+-----+
|start|count|
+-----+-----+
| 10| 2|
| 12| 2|
+-----+-----+
这篇关于PySpark 数字窗口分组依据的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文