Spark Structured Streaming 中的多重聚合 [英] Multiple aggregations in Spark Structured Streaming

问题描述

我想在 Spark Structured Streaming 中进行多次聚合.

I would like to do multiple aggregations in Spark Structured Streaming.

像这样:

• 读取输入文件流(从文件夹中)
• 执行聚合 1(带有一些转换)
• 执行聚合 2(以及更多转换)

当我在结构化流中运行它时，它给我一个错误流数据帧/数据集不支持多个流聚合".

When I run this in Structured Streaming, it gives me an error "Multiple streaming aggregations are not supported with streaming DataFrames/Datasets".

有没有办法在 Structured Streaming 中进行这种多重聚合?

Is there a way to do such multiple aggregations in Structured Streaming?

推荐答案

这个不支持，但也有其他方法.就像执行单个聚合并将其保存到 kafka 一样.从 kafka 读取它并再次应用聚合.这对我有用.

This is not supported, but there are other ways also. Like performing single aggregation and saving it to kafka. Read it from kafka and apply aggregation again. This has worked for me.

这篇关于Spark Structured Streaming 中的多重聚合的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！