调试自定义 Kafka 连接器的简单有效方法是什么? [英] What is a simple, effective way to debug custom Kafka connectors?

问题描述

我正在使用几个 Kafka 连接器，在控制台输出中没有看到它们的创建/部署有任何错误，但是我没有得到我正在寻找的结果(没有任何结果)，期望或其他).我基于 Kafka 的示例 FileStream 连接器制作了这些连接器，因此我的调试技术基于示例中使用的 SLF4J Logger 的使用.我搜索了我认为会在控制台输出中生成的日志消息，但无济于事.我是否在错误的地方寻找这些消息?或者也许有更好的方法来调试这些连接器?

I'm working a couple of Kafka connectors and I don't see any errors in their creation/deployment in the console output, however I am not getting the result that I'm looking for (no results whatsoever for that matter, desired or otherwise). I made these connectors based on Kafka's example FileStream connectors, so my debug technique was based off the use of the SLF4J Logger that is used in the example. I've searched for the log messages that I thought would be produced in the console output, but to no avail. Am I looking in the wrong place for these messages? Or perhaps is there a better way of going about debugging these connectors?

我在实现中引用的 SLF4J 记录器的使用示例:

Example uses of the SLF4J Logger that I referenced for my implementation:

Kafka FileStreamSinkTask

Kafka FileStreamSourceTask

推荐答案

我会尽量广泛地回答你的问题.一种进行连接器开发的简单方法如下:

I will try to reply to your question in a broad way. A simple way to do Connector development could be as follows:

• 通过查看众多公开可用的 Kafka 连接器之一来构建和构建连接器源代码(您可以在此处找到广泛的列表:https://www.confluent.io/product/connectors/ )
• 从 https://www.confluent 下载最新的 Confluent 开源版本 (>= 3.3.0).io/下载/

• 通过以下方式之一使您的连接器包可用于 Kafka Connect:

• Structure and build your connector source code by looking at one of the many Kafka Connectors available publicly (you'll find an extensive list available here: https://www.confluent.io/product/connectors/ )
• Download the latest Confluent Open Source edition (>= 3.3.0) from https://www.confluent.io/download/

• Make your connector package available to Kafka Connect in one of the following ways:

1. 将所有连接器 jar 文件(连接器 jar 和依赖 jar，不包括 Connect API jar)存储到文件系统中的某个位置，并通过将此位置添加到plugin.path 连接工作器属性中的属性.例如，如果您的连接器 jar 存储在 /opt/connectors/my-first-connector 中，您将在您的工作器属性中设置 plugin.path=/opt/connectors(见下文).
2. 将所有连接器 jar 文件存储在 ${CONFLUENT_HOME}/share/java 下的文件夹中.例如:${CONFLUENT_HOME}/share/java/kafka-connect-my-first-connector.(需要以 kafka-connect- 前缀开头，以便启动脚本获取).$CONFLUENT_HOME 是您安装 Confluent Platform 的位置.

1. Store all your connector jar files (connector jar plus dependency jars excluding Connect API jars) to a location in your filesystem and enable plugin isolation by adding this location to the plugin.path property in the Connect worker properties. For instance, if your connector jars are stored in /opt/connectors/my-first-connector, you will set plugin.path=/opt/connectors in your worker's properties (see below).
2. Store all your connector jar files in a folder under ${CONFLUENT_HOME}/share/java. For example: ${CONFLUENT_HOME}/share/java/kafka-connect-my-first-connector. (Needs to start with kafka-connect- prefix to be picked up by the startup scripts). $CONFLUENT_HOME is where you've installed Confluent Platform.

或者，通过将 ${CONFLUENT_HOME}/etc/kafka/connect-log4j.properties 中 Connect 的日志级别更改为 DEBUG 来增加日志记录，或者甚至 TRACE.

Optionally, increase your logging by changing the log level for Connect in ${CONFLUENT_HOME}/etc/kafka/connect-log4j.properties to DEBUG or even TRACE.

使用 Confluent CLI 启动所有服务，包括 Kafka Connect.详情请见:http://docs.confluent.io/current/connect/quickstart.html

Use Confluent CLI to start all the services, including Kafka Connect. Details here: http://docs.confluent.io/current/connect/quickstart.html

简述:confluent start

注意:CLI 当前加载的 Connect worker 的属性文件是 ${CONFLUENT_HOME}/etc/schema-registry/connect-avro-distributed.properties.如果您选择启用类加载隔离以及需要更改 Connect 工作器的属性，那么您应该编辑该文件.

Note: The Connect worker's properties file currently loaded by the CLI is ${CONFLUENT_HOME}/etc/schema-registry/connect-avro-distributed.properties. That's the file you should edit if you choose to enable classloading isolation but also if you need to change your Connect worker's properties.

• Connect worker 运行后，运行以下命令启动连接器:

  • Once you have Connect worker running, start your connector by running:

    融合负载-d

    或

    融合负载-d

    连接器配置可以是 java 属性或 JSON 格式.

    The connector configuration can be either in java properties or JSON format.

    运行confluent log connect 打开 Connect worker 的日志文件，或者通过运行直接导航到你的日志和数据的存储位置

    Run confluent log connect to open the Connect worker's log file, or navigate directly to where your logs and data are stored by running

    cd "$( confluent current )"

    注意:通过适当设置环境变量 CONFLUENT_CURRENT 来更改在 Confluent CLI 会话期间存储日志和数据的位置.例如.鉴于 /opt/confluent 存在并且是您要存储数据的位置，请运行:

    Note: change where your logs and data are stored during a session of the Confluent CLI by setting the environment variable CONFLUENT_CURRENT appropriately. E.g. given that /opt/confluent exists and is where you want to store your data, run:

    导出 CONFLUENT_CURRENT=/opt/confluent
汇合流

    • 最后，以交互方式调试连接器的一种可能方法是在使用 Confluent CLI 开始连接之前应用以下内容:

      • Finally, to interactively debug your connector a possible way is to apply the following before starting Connect with Confluent CLI :

        confluent 停止连接
export CONNECT_DEBUG=y;export DEBUG_SUSPEND_FLAG=y;
confluent 开始连接

        然后与您的调试器连接(例如远程连接到 Connect worker(默认端口:5005).要停止在调试模式下运行连接，只需运行:unset CONNECT_DEBUG; unset DEBUG_SUSPEND_FLAG;完成.

        and then connect with your debugger (for instance remotely to the Connect worker (default port: 5005). To stop running connect in debug mode, just run: unset CONNECT_DEBUG; unset DEBUG_SUSPEND_FLAG; when you are done.

        我希望以上内容能让您的连接器开发变得更轻松……更有趣！

        I hope the above will make your connector development easier and ... more fun!

        这篇关于调试自定义 Kafka 连接器的简单有效方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！