卡夫卡的背压 [英] Back pressure in Kafka

问题描述

我在卡夫卡(Kafka)遇到一种情况，生产者以比消费者消费率高得多的速度发布消息.我必须在kafka中实施反压实施，以进一步消耗和处理.

I have a situation in Kafka where the producer publishes the messages at a very higher rate than the consumer consumption rate. I have to implement the back pressure implementation in kafka for further consumption and processing.

请让我知道如何在spark和普通的Java API中实现.

Please let me know how can I implement in spark and also in normal java api.

推荐答案

Kafka在这里充当监管者.您可以按照想要的任何速率将其生成到Kafka中，从而将经纪人向外扩展以适应摄取速率.然后，您可以按照自己的意愿进行消费；Kafka保留数据并跟踪消费者在读取数据过程中的偏移量.

Kafka acts as the regulator here. You produce at whatever rate you want to into Kafka, scaling the brokers out to accommodate the ingest rate. You then consume as you want to; Kafka persists the data and tracks the offset of the consumers as they work their way through the data they read.

这篇关于卡夫卡的背压的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！