调试自定义Kafka连接器的简单有效方法是什么? [英] What is a simple, effective way to debug custom Kafka connectors?

问题描述

我正在使用几个Kafka连接器，但在控制台输出中看不到它们的创建/部署中的任何错误，但是我没有得到想要的结果(无论如何都没有结果) ，所需或其他).我基于Kafka的示例FileStream连接器制作了这些连接器，因此我的调试技术基于示例中使用的SLF4J Logger的使用.我搜索了我认为会在控制台输出中产生的日志消息，但无济于事.我在这些消息中找错了地方吗?或者也许有更好的方法来调试这些连接器?

I'm working a couple of Kafka connectors and I don't see any errors in their creation/deployment in the console output, however I am not getting the result that I'm looking for (no results whatsoever for that matter, desired or otherwise). I made these connectors based on Kafka's example FileStream connectors, so my debug technique was based off the use of the SLF4J Logger that is used in the example. I've searched for the log messages that I thought would be produced in the console output, but to no avail. Am I looking in the wrong place for these messages? Or perhaps is there a better way of going about debugging these connectors?

我为实现所引用的SLF4J Logger的示例用法:

Example uses of the SLF4J Logger that I referenced for my implementation:

Kafka FileStreamSinkTask

Kafka FileStreamSourceTask

推荐答案

我将尝试广泛地回答您的问题.进行连接器开发的简单方法如下:

I will try to reply to your question in a broad way. A simple way to do Connector development could be as follows:

• 通过查看公开提供的许多Kafka连接器之一来构建和构建连接器源代码(您可以在此处找到广泛的列表: https://www.confluent下载最新的Confluent开源版本(> = 3.3.0). io/download/

• 通过以下其中一种方式使连接器包可用于Kafka Connect:

• Structure and build your connector source code by looking at one of the many Kafka Connectors available publicly (you'll find an extensive list available here: https://www.confluent.io/product/connectors/ )
• Download the latest Confluent Open Source edition (>= 3.3.0) from https://www.confluent.io/download/

• Make your connector package available to Kafka Connect in one of the following ways:

1. 将所有连接器jar文件(连接器jar以及依赖关系jar(不包括Connect API jar)除外)存储到文件系统中的某个位置，并将此位置添加到文件系统中以启用插件隔离 连接工作器属性中的plugin.path属性.例如，如果连接器罐子存储在/opt/connectors/my-first-connector中，则将在工作人员的属性中设置plugin.path=/opt/connectors(请参见下文).
2. 将所有连接器jar文件存储在${CONFLUENT_HOME}/share/java下的文件夹中.例如:${CONFLUENT_HOME}/share/java/kafka-connect-my-first-connector. (需要以kafka-connect-前缀开头，以由启动脚本选择). $ CONFLUENT_HOME是您安装Confluent平台的地方.

1. Store all your connector jar files (connector jar plus dependency jars excluding Connect API jars) to a location in your filesystem and enable plugin isolation by adding this location to the plugin.path property in the Connect worker properties. For instance, if your connector jars are stored in /opt/connectors/my-first-connector, you will set plugin.path=/opt/connectors in your worker's properties (see below).
2. Store all your connector jar files in a folder under ${CONFLUENT_HOME}/share/java. For example: ${CONFLUENT_HOME}/share/java/kafka-connect-my-first-connector. (Needs to start with kafka-connect- prefix to be picked up by the startup scripts). $CONFLUENT_HOME is where you've installed Confluent Platform.

(可选)通过将${CONFLUENT_HOME}/etc/kafka/connect-log4j.properties中的Connect的日志级别更改为DEBUG甚至TRACE来增加日志记录.

Optionally, increase your logging by changing the log level for Connect in ${CONFLUENT_HOME}/etc/kafka/connect-log4j.properties to DEBUG or even TRACE.

使用Confluent CLI启动所有服务，包括Kafka Connect.此处的详细信息: http://docs.confluent.io/current/connect/quickstart.html

Use Confluent CLI to start all the services, including Kafka Connect. Details here: http://docs.confluent.io/current/connect/quickstart.html

简而言之:confluent start

注意:CLI当前加载的Connect worker的属性文件是${CONFLUENT_HOME}/etc/schema-registry/connect-avro-distributed.properties.如果您选择启用类加载隔离，但也需要更改Connect worker的属性，则应编辑该文件.

Note: The Connect worker's properties file currently loaded by the CLI is ${CONFLUENT_HOME}/etc/schema-registry/connect-avro-distributed.properties. That's the file you should edit if you choose to enable classloading isolation but also if you need to change your Connect worker's properties.

• 一旦运行了Connect worker，请运行以下命令来启动连接器:

  • Once you have Connect worker running, start your connector by running:

    confluent load <connector_name> -d <connector_config.properties>

    或

    confluent load <connector_name> -d <connector_config.json>

    连接器配置可以采用Java属性或JSON格式.

    The connector configuration can be either in java properties or JSON format.

    运行 confluent log connect打开Connect worker的日志文件，或通过运行直接导航到您的日志和数据的存储位置

    Run confluent log connect to open the Connect worker's log file, or navigate directly to where your logs and data are stored by running

    cd "$( confluent current )"

    注意:通过适当设置环境变量CONFLUENT_CURRENT，可以在Confluent CLI会话期间更改日志和数据的存储位置.例如.鉴于/opt/confluent存在并且是您要存储数据的位置，请运行:

    Note: change where your logs and data are stored during a session of the Confluent CLI by setting the environment variable CONFLUENT_CURRENT appropriately. E.g. given that /opt/confluent exists and is where you want to store your data, run:

    export CONFLUENT_CURRENT=/opt/confluent
confluent current

    • 最后，要交互式地调试连接器，一种可能的方法是在使用Confluent CLI启动Connect之前应用以下内容:

      • Finally, to interactively debug your connector a possible way is to apply the following before starting Connect with Confluent CLI :

        confluent stop connect
export CONNECT_DEBUG=y; export DEBUG_SUSPEND_FLAG=y;
confluent start connect

        confluent stop connect
export CONNECT_DEBUG=y; export DEBUG_SUSPEND_FLAG=y;
confluent start connect

        ，然后与调试器连接(例如，远程连接到Connect worker(默认端口:5005).要停止在调试模式下运行连接，只需在完成后运行:unset CONNECT_DEBUG; unset DEBUG_SUSPEND_FLAG;.

        and then connect with your debugger (for instance remotely to the Connect worker (default port: 5005). To stop running connect in debug mode, just run: unset CONNECT_DEBUG; unset DEBUG_SUSPEND_FLAG; when you are done.

        我希望以上内容将使您的连接器开发更加轻松，并且……更加有趣！

        I hope the above will make your connector development easier and ... more fun!

        这篇关于调试自定义Kafka连接器的简单有效方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！