Spark结构化流中的多个聚合 [英] Multiple aggregations in Spark Structured Streaming

问题描述

我想在Spark结构化流中进行多个聚合.

I would like to do multiple aggregations in Spark Structured Streaming.

类似这样的东西:

• 从文件夹中读取输入文件流
• 执行聚合1(进行一些转换)
• 执行聚合2(以及更多转换)

当我在结构化流中运行它时，它给我一个错误流数据帧/数据集不支持多个流聚合".

When I run this in Structured Streaming, it gives me an error "Multiple streaming aggregations are not supported with streaming DataFrames/Datasets".

有没有一种方法可以在结构化流媒体中进行这样的多重聚合?

Is there a way to do such multiple aggregations in Structured Streaming?

推荐答案

不支持此方法，但是还有其他方法.就像执行单个聚合并将其保存到kafka一样.从kafka读取它，然后再次应用聚合.这对我有用.

This is not supported, but there are other ways also. Like performing single aggregation and saving it to kafka. Read it from kafka and apply aggregation again. This has worked for me.

这篇关于Spark结构化流中的多个聚合的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！