Kafka Connector - 分布式 - 负载均衡任务 [英] Kafka Connector - distributed - load balancing tasks

问题描述

我正在运行 Confluent Kafka 的开发环境，Windows 上的社区版，版本 3.0.1-2.11.我正在尝试在 2 个连接器实例之间实现任务的负载平衡.我在同一台机器上运行 Kafka Zookepper、服务器、REST 服务和 2 个 Connect 实例.连接器的属性文件之间的唯一区别是其余端口，因为它们在同一台机器上运行.我不会为连接器偏移、配置、状态创建主题.我是不是该?我有接收器连接器的自定义代码.

I am running development environment for Confluent Kafka, Community edition on Windows, version 3.0.1-2.11. I am trying to achieve load balancing of tasks between 2 instances of connector. I am running Kafka Zookepper, Server, REST services and 2 instance of Connect distributed on the same machine. Only difference between properties file for connectors is rest port since they are running on the same machine. I don't create topics for connector offsets, config, status. Should I? I have custom code for sink connector.

当我为接收器连接器创建工作器时，我通过执行 POST 请求来完成此操作

When I create worker for my sink connector I do this by executing POST request

POST http://localhost:8083/connectors

朝向任何正在运行的连接器.检查是否在 URL 处完成加载的工作

toward any of the running connectors. Checking is there loaded worker is done at URL

GET http://localhost:8083/connectors

我的接收器连接器在代码中有 System.out.println() 行，我可以使用它来跟踪控制台日志中代码的输出.当我的工作人员正在运行时，我可以看到只有一个连接器实例正在执行代码.如果我终止一个连接器，另一个实例将接管工作程序并恢复执行.然而这不是我想要的.我的目标是两个连接器实例都在运行工作代码，以便它们可以在它们之间共享负载.我试图克服一些开源连接器，看看是否有编写连接器代码的细节，但没有成功.

My sink connector has System.out.println() lines in code with which I can follow output of my code in the console log. When my worker is running I can see that only one instance of connector is executing code. If I terminate one connector another instance will take over the worker and execution will resume. However this is not what I want. My goal is that both connector instances are running worker code so that they can share the load between them. I've tried to got over some open source connectors to see is there specifics in writing code of connectors but with no success.

为了解决这个问题，我做了一些不同的尝试，但都没有成功.我可以重写我的业务代码来解决这个问题，但我很确定我错过了一些对我来说并不明显的东西.最近我评论了 Robin Moffatt 对这个问题的回答.

I've made some different attempts to tackle this problem but with no success. I could rewrite my business code to come around this but I'm pretty sure I'm missing on something not obvious for me. Recently I commented on Robin Moffatt's answer of this question.

推荐答案

从它的声音来看，您的自定义代码没有正确生成您期望的任务数量.

From the sounds of it your custom code is not correctly spawning the number of tasks that you are expecting.

• 确保您已在配置中设置 tasks.max >1
• 确保您的连接器正确地为 taskConfigs
创建了适当数量的任务

• Make sure that you've set tasks.max >1 in your config
• Make sure that your connector is correctly creating the appropriate number of tasks to taskConfigs

参考资料:

• https://opencredo.com/blogs/kafka-connect-source-connectors-a-detailed-guide-to-connecting-to-what-you-love/
• https://docs.confluent.io/current/connect/devguide.html
• https://enfuse.io/a-diy-guide-to-kafka-connectors/

这篇关于Kafka Connector - 分布式 - 负载均衡任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！