如何在 flink 上使用 Ratelimiter? [英] How to use Ratelimiter on flink?

查看:200
本文介绍了如何在 flink 上使用 Ratelimiter?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我想在我的 Flink 作业中限制 Kafka 消费者.

I want to throttle the Kafka consumers in my Flink job.

查看 Flink 1.12 的源代码,我找到了 FlinkConnectorRateLimiterGuavaFlinkConnectorRateLimiter.但是我找不到任何将这个速率限制器连接到 FlinkKafkaConsumer 的东西.

Looking through the source code for Flink 1.12, I find FlinkConnectorRateLimiter and GuavaFlinkConnectorRateLimiter. But I can't find anything connecting this rate limiter to FlinkKafkaConsumer.

如何在 Flink 1.12 中实现 Kafka 的速率限制?

How can I implement rate limiting for Kafka in Flink 1.12?

推荐答案

FlinkConnectorRateLimiter 在 Flink 1.12 中被删除的遗留 Kafka 消费者 (flink-connector-kafka-0.10) 中可用.当前 kafka 消费者不提供速率限制.

FlinkConnectorRateLimiter was available with the legacy Kafka consumer (flink-connector-kafka-0.10), which was dropped in Flink 1.12. The current kafka consumer does not offer rate limiting.

请参阅此邮件列表线程 -- http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Rate-Limit-in-FlinkConsumer-td36417.html -- 一些讨论.简而言之,一旦对背压和事件时间偏差下的检查点进行的改进完成后,速率限制应该不再具有任何吸引力,因此实际上没有任何兴趣增加对速率限制的支持.

See this mailing list thread -- http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Rate-Limit-in-FlinkConsumer-td36417.html -- for some discussion. In short, rate limiting should no longer have any appeal once the in-progress improvements to checkpointing under backpressure and event-time skew are completed, so there isn't really any appetite to add support for rate limiting.

但是,上面的邮件列表线程确实包含一个示例,展示了如何通过扩展 FlinkKafkaConsumer 来覆盖 emitRecordemitRecordWithTimestamp.

However, the mailing list thread above does include an example showing how to implement rate limiting for Kafka yourself by extending FlinkKafkaConsumer to override emitRecord and emitRecordWithTimestamp.

这篇关于如何在 flink 上使用 Ratelimiter?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆