在多个节点上运行kafka connect分布式模式 [英] Run kafka connect distributed mode on many nodes

问题描述

我正在测试一个 kafka 连接器的弹性，我想在它运行时杀死一个工人，从而杀死连接器实例.最简单的方法可能是强制分布式模式在多个节点上运行，然后杀死该节点上的工作进程(对吧?).我怎样才能让 Kafka 连接在不仅仅是它启动的节点上产生工作人员?这是在工作配置中定义的吗?

I'm resiliency testing a kafka connector and I'd like to kill off a worker while it's running, thus killing the connector instance. The easiest way is probably going to be to force distributed mode to run over more than one node, then just kill the worker process on that node (right?). How can I make Kafka connect spawn workers on more than just the node it's started on? Is this something which is defined in worker config?

推荐答案

所以最后我做的是:

• 将 Kafka Connect 分布式模式所需的所有 jar 复制到我想要运行它的两个节点上(在 HDP 2.5.3 中，您只能在一个节点上获取这些 jar).
• 在两个节点上，我都运行了带有指向我的 jar 的属性文件的启动脚本.
• 使用 REST 接口，我发布了带有任务的连接器，我可以看到一个工作人员拥有连接器实例，另一个工作人员拥有其任务.
• 我杀死了任务工作节点(使用 ps -ef | grep connect)，并看到它在剩余的节点上重新生成.
• 我重置了测试并尝试关闭连接器实例节点，令我惊讶的是，连接器实例在另一个节点上重新启动.

• Copied all the jars I needed for Kafka Connect distributed mode to the two nodes I wanted to run it on (in HDP 2.5.3 you only get those jars on one node).
• On both nodes, I ran the start script with properties file pointing to my jars.
• Using the REST interface I posted my connector with a task, and I could see that one worker had the connector instance and another had its task.
• I killed off the task worker node (using ps -ef | grep connect), and saw that it had respawned on the remaining node.
• I reset the test and tried killing off the connector instance node, and to my amazement, the connector instance restarted on the other node.

总结一下我的弹性测试，Kafka Connect 好像在玩打地鼠游戏；你可以在任何地方杀死任务或连接器，它们只会在其他地方重生.

In summary of my resiliency testing, Kafka Connect seems to be like playing whack-a-mole; you can kill off tasks or connectors wherever they are, and they will just respawn somewhere else.

这篇关于在多个节点上运行kafka connect分布式模式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！