卡夫卡的背压 [英] Back pressure in Kafka

问题描述

我在 Kafka 中遇到过这样一种情况，即生产者以比消费者消费率高得多的速度发布消息.我必须在kafka中实现背压实现，以便进一步消费和处理.

I have a situation in Kafka where the producer publishes the messages at a very higher rate than the consumer consumption rate. I have to implement the back pressure implementation in kafka for further consumption and processing.

请让我知道如何在 spark 和普通的 java api 中实现.

Please let me know how can I implement in spark and also in normal java api.

推荐答案

Kafka 在这里充当监管者.你可以以任何你想要的速度在 Kafka 中生成，扩展代理以适应摄取率.然后你可以随心所欲地消费；Kafka 会保留数据并跟踪消费者在处理他们读取的数据时的偏移量.

Kafka acts as the regulator here. You produce at whatever rate you want to into Kafka, scaling the brokers out to accommodate the ingest rate. You then consume as you want to; Kafka persists the data and tracks the offset of the consumers as they work their way through the data they read.

这篇关于卡夫卡的背压的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！