Spark结构化流如何处理背压? [英] How Spark Structured Streaming handles backpressure?

问题描述

我正在分析Spark结构化流上的背压功能.有人知道细节吗?是否可以通过代码调整处理传入记录? 谢谢

I'm analyzing the backpressure feature on Spark Structured Streaming. Does anyone know the details? Is it possible to tune process incoming records by code? Thanks

推荐答案

如果要在结构化流中动态更改每个内部批处理的大小，请否.结构化流中没有基于接收器的源，因此完全没有必要.从另一个角度来看，结构化流无法进行真正的背压，因为例如Spark无法告知其他应用程序减慢将数据推入Kafka的速度.

If you mean dynamically changing the size of each internal batch in Structured Streaming, then NO. There are not receiver-based sources in Structured Streaming, so that's totally not necessary. From another point of view, Structured Streaming cannot do real backpressure, because, such as, Spark cannot tell other applications to slow down the speed of pushing data into Kafka.

通常，结构化流默认会尝试尽快处理数据.每个源中都有允许控制处理速率的选项，例如File源中的maxFilesPerTrigger和Kafka源中的maxOffsetsPerTrigger.阅读以下链接以获取更多详细信息:

Generally, Structured Streaming will try to process data as fast as possible by default. There are options in each source to allow to control the processing rate, such as maxFilesPerTrigger in File source, and maxOffsetsPerTrigger in Kafka source. Read the following links for more details:

http://spark.apache. org/docs/latest/structured-streaming-programming-guide.html#input-sources http://spark.apache.org/docs/latest/structured -streaming-kafka-integration.html

这篇关于Spark结构化流如何处理背压?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！