在许多节点上运行kafka connect分布式模式 [英] Run kafka connect distributed mode on many nodes

问题描述

我正在对kafka连接器进行弹性测试，我想杀死一个正在运行的工作程序，从而杀死该连接器实例.最简单的方法可能是强制分布式模式在一个以上的节点上运行，然后杀死该节点上的工作进程(对吗?).如何使Kafka不仅在其启动的节点上连接spawn worker?这是工作程序配置中定义的吗?

I'm resiliency testing a kafka connector and I'd like to kill off a worker while it's running, thus killing the connector instance. The easiest way is probably going to be to force distributed mode to run over more than one node, then just kill the worker process on that node (right?). How can I make Kafka connect spawn workers on more than just the node it's started on? Is this something which is defined in worker config?

推荐答案

所以最后我做了:

• 将我将Kafka Connect分布式模式所需的所有jar复制到我要在其上运行的两个节点上(在HDP 2.5.3中，您只能将这些jar放在一个节点上).
• 在两个节点上，我都运行了带有指向我的jar的属性文件的启动脚本.
• 使用REST接口，我向连接器发布了一个任务，我可以看到一个工作人员拥有该连接器实例，另一个工作人员具有其任务.
• 我杀死了Task Worker节点(使用 ps -ef | grep connect )，并看到它在其余节点上重新生成.
• 我重置了测试并尝试终止连接器实例节点，但令我惊讶的是，连接器实例在另一个节点上重新启动.

• Copied all the jars I needed for Kafka Connect distributed mode to the two nodes I wanted to run it on (in HDP 2.5.3 you only get those jars on one node).
• On both nodes, I ran the start script with properties file pointing to my jars.
• Using the REST interface I posted my connector with a task, and I could see that one worker had the connector instance and another had its task.
• I killed off the task worker node (using ps -ef | grep connect), and saw that it had respawned on the remaining node.
• I reset the test and tried killing off the connector instance node, and to my amazement, the connector instance restarted on the other node.

在我的弹性测试的总结中，Kafka Connect似乎就像在打w子.您可以杀死任务或连接器，无论它们在哪里，它们都将在其他地方重新生成.

In summary of my resiliency testing, Kafka Connect seems to be like playing whack-a-mole; you can kill off tasks or connectors wherever they are, and they will just respawn somewhere else.

这篇关于在许多节点上运行kafka connect分布式模式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！