分析异常：流式数据帧/数据集不支持非基于时间的窗口；；尽管存在基于时间的窗口 [英] org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;; despite of time-based window

问题描述

我正在对电光结构化流进行窗口排序：

val filterWindow: WindowSpec = Window
  .partitionBy("key")
  .orderBy($"time")

controlDataFrame=controlDataFrame.withColumn("Make Coffee", $"value").    
  withColumn("datetime", date_trunc("second", current_timestamp())).
  withColumn("time", current_timestamp()).
  withColumn("temp_rank", rank().over(filterWindow))
  .filter(col("temp_rank") === 1)
  .drop("temp_rank").
  withColumn("digitalTwinId", lit(digitalTwinId)).
  withWatermark("datetime", "10 seconds")

我正在获取timeAscurrent_timestamp()，在架构中我看到其类型为StructField(time,TimestampType,true)

为什么电光3.0不允许我基于它进行窗口操作，只有以下几个例外，因为字段明显是按时间填写的？

21/11/22 10:34:03 WARN SparkSession$Builder: Using an existing SparkSession; some spark core configurations may not take effect.

org.apache.spark.sql.AnalysisException: Non-time-based windows are not supported on streaming DataFrames/Datasets;;
Window [rank(time#163) windowspecdefinition(key#150, time#163 ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS temp_rank#171], [key#150], [time#163 ASC NULLS FIRST]
+- Project [key#150, value#151, Make Coffee#154, datetime#158, time#163]

推荐答案

so，根据答案https://issues.apache.org/jira/browse/SPARK-37439?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel

基于时间的窗口不只是基于时间戳格式的列的Window/WindowSpec。您应该对时间戳列显式使用session_window()函数。

http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#types-of-time-windows

这篇关于分析异常：流式数据帧/数据集不支持非基于时间的窗口；；尽管存在基于时间的窗口的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！