结构化流例外:流聚合不支持追加输出模式 [英] Structured Streaming exception: Append output mode not supported for streaming aggregations

问题描述

运行火花作业时出现以下错误:

I am getting the following error when I run my spark job:

org.apache.spark.sql.AnalysisException:当流数据帧/数据集上存在流聚合时，不支持追加输出模式；

org.apache.spark.sql.AnalysisException: Append output mode not supported when there are streaming aggregations on streaming DataFrames/DataSets;;

我不确定是否由于缺少

I am not sure if the issue is being caused due to lack of a watermark,which I don't know how to apply in this context. Following is the aggregation operation applied:

def aggregateByValue(): DataFrame = {
  df.withColumn("Value", expr("(BookingClass, Value)"))
    .groupBy("AirlineCode", "Origin", "Destination", "PoS", "TravelDate", "StartSaleDate", "EndSaleDate", "avsFlag")
    .agg(collect_list("Value").as("ValueSeq"))
    .drop("Value")
}

用法:

val theGroupedDF = theDF
  .multiplyYieldByHundred
  .explodeDates
  .aggregateByValue

val query = theGroupedDF.writeStream
  .outputMode("append")
  .format("console")
  .start()
query.awaitTermination()

推荐答案

将 outputMode 更改为 complete 即可解决此问题.

Changing the outputMode to complete solved the issue.

val query = theGroupedDF.writeStream
  .outputMode("complete")
  .format("console")
  .start()
query.awaitTermination()

这篇关于结构化流例外:流聚合不支持追加输出模式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！