Kafka 连接集群设置或启动连接工作器 [英] Kafka connect cluster setup or launching connect workers
问题描述
我正在通过 kafka connect，我正在尝试了解概念.
I am going through kafka connect, and i am trying to get the concepts.
假设我已经设置了 kafka 集群(节点 k1、k2 和 k3)并且它正在运行，现在我想以分布式模式在不同的节点上运行 kafka connect worker，比如说 c1 和 c2.
Let us say I have kafka cluster (nodes k1, k2 and k3) setup and it is running, now i want to run kafka connect workers in different nodes say c1 and c2 in distributed mode.
几个问题.
1) 要以分布式模式运行或启动 kafka 连接，我需要使用命令 ../bin/connect-distributed.sh，该命令在 kakfa 集群节点中可用，因此我需要启动kafka 从任何一个 kafka 集群节点连接?或者我启动 kafka connect 的任何节点都需要有 kafka 二进制文件，以便我能够使用 ../bin/connect-distributed.sh
1) To run or launch kafka connect in distributed mode I need to use command ../bin/connect-distributed.sh, which is available in kakfa cluster nodes, so I need to launch kafka connect from any one of the kafka cluster nodes? or any node from where I launch kafka connect needs to have kafka binaries so that i will be able to use ../bin/connect-distributed.sh
2) 我需要将我的连接器插件复制到我执行第 1 步的任何 kafka 集群节点(或所有集群节点?)?
2) I need to copy the my connector plugins to any kafka cluster node( or to all cluster nodes?) from where I do the step 1?
3) 在工作节点上启动 jvm 进程之前，kafka 如何将这些连接器插件复制到工作节点?因为插件是有我的任务代码的插件，它需要复制到工作人员才能在工作人员中启动进程.
3) how does kafka copies these connector plugins to worker node before starting jvm process on the worker node? because the plugin is the one which has my task code and it needs to be copied to worker in order to start the process in worker.
4) 我是否需要在连接集群节点 c1 和 c2 中安装任何东西，比如需要安装 java 或任何与 kafka 连接相关的东西?
4) Do i need to install anything in connect cluster nodes c1 and c2, like need to install java or any kafka connect related?
5) 在某些地方它说使用融合平台，但我想先单独使用 apache kafka connect 启动它.
5) In some places it says use confluent platform but i would like to start it with apache kafka connect alone first.
有人可以通过一些光线甚至指向某些资源的指针也会有所帮助.
can some one please through some light or even pointer to some resources would also help.
谢谢.
推荐答案
1) 为了获得高可用的 kafka-connect 服务，您需要运行至少两个 connect-distributed.sh 在两台具有相同 group.id 的不同机器上.您可以在此处找到有关每个工作人员配置的更多详细信息.为了提高性能，Connect 应该独立于代理和 Zookeeper 机器运行.
1) In order to have a highly available kafka-connect service you need to run at least two instances of connect-distributed.sh on two distinct machines that have the same group.id. You can find more details regarding the configuration of each worker here. For improved performance, Connect should be ran independently of the broker and Zookeeper machines.
2) 是的，您需要将所有连接器放在您计划的每台机器上的 plugin.path 下(通常在 /usr/share/java/ 下)运行卡夫卡连接.
2) Yes, you need to place all your connectors under plugin.path (normally under /usr/share/java/) on every machine that you are planning to run kafka-connect.
3) kafka-connect 将在启动时加载连接器.你不需要处理这个.请注意，如果您的 kafka-connect 实例正在运行并且添加了新的连接器，则需要重新启动该服务.
3) kafka-connect will load the connectors on startup. You don't need to handle this. Note that if your kafka-connect instance is running and a new connector is added, you need to restart the service.
4) 您需要在所有机器上安装 Java.特别是对于 Confluent 平台:
4) You need to have Java installed on all your machines. For Confluent Platform particularly:
此版本的 Confluent Platform 支持 Java 1.7 和 1.8(目前不支持 Java 1.9).你应该运行垃圾优先 (G1) 垃圾收集器.有关更多信息，请参阅支持的版本和互操作性.
Java 1.7 and 1.8 are supported in this version of Confluent Platform (Java 1.9 is currently not supported). You should run with the Garbage-First (G1) garbage collector. For more information, see the Supported Versions and Interoperability.
5) 视情况而定.Confluent 由 Apache Kafka 的原始创建者创立，它是一个更完整的发行版，添加了模式管理、连接器和客户端.它还带有 KSQL，如果您需要对某些事件采取行动，这将非常有用.Confluent 只是在 Apache Kafka 发行版之上添加，它不是修改版本.
5) It depends. Confluent was founded by the original creators of Apache Kafka and it comes as a more complete distribution adding schema management, connectors and clients. It also comes with KSQL which is quite useful if you need to act on certain events. Confluent simply adds on top of the Apache Kafka distribution, it's not a modified version.
这篇关于Kafka 连接集群设置或启动连接工作器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
