手动安装Kafka连接器 [英] Install Kafka connector manually

问题描述

我正在尝试安装 twitter 源连接器并找到以下链接 link1 和 link2.第一个展示了如何安装 twitter 源连接器，第二个展示了如何手动安装连接器(始终是 Confluent Platform).

I'm trying install twitter source connector and found the following links link1 and link2. The first one shows how to install twitter source connector and the second generally how to install manually a connector (always Confluent Platform).

但是我对整个过程感到困惑.我遵循了 此处，到目前为止它工作正常.但是，由于 twitter 源连接器不在连接器中心的列表中，因此需要手动安装.根据link1，我在某处克隆了一个git repo - 问题:我在哪里克隆它?在 docker-compose.yml 中有一堆服务.特别是，在我启动 yaml 后的 docker-compose ps 显示

However I'm confused from the entire procedure. I followed the steps in here, and so far it's working fine. However since the twitter source connector is not in the list of the Connector Hub it needs to be installed manually. According to link1, I clone a git repo somewhere - Question: where do I clone it? In the docker-compose.yml there are a bunch of services. In particular, a docker-compose ps after I spin up the yaml shows

       Name                    Command                  State                         Ports
  ------------------------------------------------------------------------------------------------------------

   broker            /etc/confluent/docker/run        Up             0.0.0.0:9092->9092/tcp
   connect           /etc/confluent/docker/run        Up             0.0.0.0:8083->8083/tcp, 9092/tcp
   control-center    /etc/confluent/docker/run        Up             0.0.0.0:9021->9021/tcp
   ksql-datagen      bash -c echo Waiting for K ...   Up
   ksqldb-cli        /bin/sh                          Up
   ksqldb-server     /etc/confluent/docker/run        Up (healthy)   0.0.0.0:8088->8088/tcp
   rest-proxy        /etc/confluent/docker/run        Up             0.0.0.0:8082->8082/tcp
   schema-registry   /etc/confluent/docker/run        Up             0.0.0.0:8081->8081/tcp
   zookeeper         /etc/confluent/docker/run        Up             0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

我不明白我应该对上面的哪一个执行link1的说明.是在动物园管理员、经纪人还是在连接中?欢迎任何帮助，如果有人有时间更彻底地围绕link1中的步骤写几句话，请随意.干杯

I don't understand to which one of the above I should carry out the instructions of link1. Is it in the zookeeper, the broker or in the connect? Any help is welcomed, please if someone has time to write out a few words around the steps in link1 more thoroughly, please feel free. Cheers

推荐答案

连接器安装在 Kafka Connect 工作线程中.它们必须在 worker 启动之前安装.

Connectors are installed in Kafka Connect workers. They must be installed before the worker is started.

该博客文章有一个更新版本，在这里:https://www.confluent.io/blog/stream-processing-twitter-data-with-ksqldb.与此相关的是一个新的演示，您可以在这里找到:https://github.com/confluentinc/demo-scene/tree/master/ksqldb-twitter

There is a newer version of that blog article, here: https://www.confluent.io/blog/stream-processing-twitter-data-with-ksqldb. Associated with that is a new demo, which you can find here: https://github.com/confluentinc/demo-scene/tree/master/ksqldb-twitter

作为演示的一部分，它安装了 twitter 连接器 在运行时.它通过覆盖容器映像的默认命令来实现这一点，而是在运行连接工作器之前运行连接器安装:

As part of the demo, it installs the twitter connector at runtime. It does this by overriding the default command of the container image and instead runs the connector installation before running the connect worker:

    command: 
      - bash 
      - -c 
      - |
        echo "Installing connector plugins"
        confluent-hub install --no-prompt jcustenborder/kafka-connect-twitter:0.3.33
        #
        echo "Launching Kafka Connect worker"
        /etc/confluent/docker/run & 
        #
        sleep infinity

这篇关于手动安装Kafka连接器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！