如何在 flink 上使用 Ratelimiter? [英] How to use Ratelimiter on flink?

问题描述

我想在我的 Flink 作业中限制 Kafka 消费者.

I want to throttle the Kafka consumers in my Flink job.

查看 Flink 1.12 的源代码，我找到了 FlinkConnectorRateLimiter 和 GuavaFlinkConnectorRateLimiter.但是我找不到任何将这个速率限制器连接到 FlinkKafkaConsumer 的东西.

Looking through the source code for Flink 1.12, I find FlinkConnectorRateLimiter and GuavaFlinkConnectorRateLimiter. But I can't find anything connecting this rate limiter to FlinkKafkaConsumer.

如何在 Flink 1.12 中实现 Kafka 的速率限制?

How can I implement rate limiting for Kafka in Flink 1.12?

推荐答案

FlinkConnectorRateLimiter 在 Flink 1.12 中被删除的遗留 Kafka 消费者 (flink-connector-kafka-0.10) 中可用.当前 kafka 消费者不提供速率限制.

FlinkConnectorRateLimiter was available with the legacy Kafka consumer (flink-connector-kafka-0.10), which was dropped in Flink 1.12. The current kafka consumer does not offer rate limiting.

请参阅此邮件列表线程 -- http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Rate-Limit-in-FlinkConsumer-td36417.html -- 一些讨论.简而言之，一旦对背压和事件时间偏差下的检查点进行的改进完成后，速率限制应该不再具有任何吸引力，因此实际上没有任何兴趣增加对速率限制的支持.

See this mailing list thread -- http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Rate-Limit-in-FlinkConsumer-td36417.html -- for some discussion. In short, rate limiting should no longer have any appeal once the in-progress improvements to checkpointing under backpressure and event-time skew are completed, so there isn't really any appetite to add support for rate limiting.

但是，上面的邮件列表线程确实包含一个示例，展示了如何通过扩展 FlinkKafkaConsumer 来覆盖 emitRecord 和 emitRecordWithTimestamp.

However, the mailing list thread above does include an example showing how to implement rate limiting for Kafka yourself by extending FlinkKafkaConsumer to override emitRecord and emitRecordWithTimestamp.

这篇关于如何在 flink 上使用 Ratelimiter?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！