带有 JdbcConnectionSource 连接器的 Kafka Connect 无法创建任务(连接器正在运行但任务不是) [英] Kafka Connect with a JdbcConnectionSource connector fails to create task (connector is RUNNING but task is not)

问题描述

我似乎经常根据查询从 JdbcConnectionSource 创建一个 Kafka Connect 连接器，并且连接器创建成功，状态为RUNNING"，但没有创建任何任务.查看我的容器的控制台日志，我没有看到任何错误的迹象，我可以说:没有错误，没有警告，没有解释任务失败的原因.我可以让其他连接器工作，但有时不能.

It seems like rather often I create a Kafka Connect connector from the JdbcConnectionSource based on a query, and the connector is created successsfully with status "RUNNING", but no task is created. Looking in the console logs of my container, I see no indication that anything is wrong that I can tell: no errors, no warnings, no explanation of why the task failed. I can get other connectors to work, but sometimes one doesn't.

当连接器无法创建 RUNNING 任务时，如何获得更多信息以进行故障排除?

我将在下面发布我的连接器配置示例.

I'll post an example of my connector config below.

我使用的是 Kafka Connect 5.4.1-ccs.

I am using Kafka Connect 5.4.1-ccs.

连接器配置(它是 JDBC 背后的 Oracle 数据库):

Connector config (it is an Oracle Database behind JDBC):

{
    "name": "FiscalYear",
    "config": {
        "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
        "tasks.max": 1,
        "connection.url": "jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=myhost.example.com)(PORT=1521))(LOAD_BALANCE=OFF)(FAILOVER=OFF)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=MY_DB_PRI)(UR=A)))",
        "connection.user":"myuser",
        "connection.password":"mypass",
        "mode": "timestamp",
        "timestamp.column.name": "MAINT_TS",
        "topic.prefix": "MyTeam.MyTopicName",
        "poll.interval.ms": 5000,
        "value.converter" : "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "numeric.mapping": "best_fit",

        "_comment": "The query is wrapped in `select * from ()` so that JdbcSourceConnector can automatically append a WHERE clause.",
        "query": "SELECT * FROM (SELECT fy_nbr, min(fy_strt_dt) fy_strt_dt, max(fy_end_dt) fy_end_dt FROM myuser.fsc_dt fd WHERE fd.fy_nbr >= 2020 and fd.fy_nbr < 2022 group by fy_nbr)/* outer query must have no WHERE clause so that the source connector can append one of its own */"
    }
}  

以及创建我的工人的 Dockerfile:

And the Dockerfile that creates my worker:

FROM confluentinc/cp-kafka-connect:latest

# each "CONNECT_" env var refers to a Kafka Connect setting; e.g. CONNECT_REST_PORT refers to setting rest.port
#  see also https://docs.confluent.io/current/connect/references/allconfigs.html

ENV CONNECT_BOOTSTRAP_SERVERS="d.mybroker.example.com:9092"
ENV CONNECT_REST_PORT="8083"
ENV CONNECT_GROUP_ID="MyGroup2" 

ENV CONNECT_CONFIG_STORAGE_TOPIC="MyTeam.ConnectorConfig" 
ENV CONNECT_OFFSET_STORAGE_TOPIC="MyTeam.ConnectorOffsets" 
ENV CONNECT_STATUS_STORAGE_TOPIC="MyTeam.ConnectorStatus" 

ENV CONNECT_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" 
ENV CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" 

ENV CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter"  
ENV CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" 

ENV CONNECT_LOG4J_ROOT_LOGLEVEL="INFO"

COPY ojdbcDrivers /usr/share/java/kafka-connect-jdbc

(我还通过我的 Helm 图表设置了 REST 广告主机名环境变量，所以这就是上面没有设置的原因.)

(I also set the REST advertised hostname environment variable via my Helm chart, so that's why it isn't set in the above.)

在它启动后，我创建了连接器，然后从 REST/status"获取它:

After it spins up, I create the connector, then get this from the REST "/status":

{"name":"FiscalYear","connector":{"state":"RUNNING","worker_id":"10.1.2.3:8083"},"tasks":[],"type":"source"}

推荐答案

当连接器无法创建 RUNNING 任务时，如何获得更多信息以进行故障排除?

How can one get more information to troubleshoot when a connector fails to create a RUNNING task?

我会提高您的 Kafka Connect 工作人员的日志记录级别.由于您使用的是 Apache Kafka 2.4，因此您可以动态执行此操作，这非常有用.向您的 Kafka Connect 工作人员发出此 REST API 调用:

I would increase the logging level on your Kafka Connect worker. Since you're using Apache Kafka 2.4 you can do this dynamically, which is rather useful. Issue this REST API call to your Kafka Connect worker:

curl -X PUT http://localhost:8083/admin/loggers/io.confluent \
     -H "Content-Type:application/json" -d '{"level": "TRACE"}'

这会将任何 Confluent 连接器的所有消息都添加到 TRACE.它还返回单个记录器的列表，您可以从中挑选不同的记录器并根据需要提高或降低它们的特定日志级别.例如:

This bumps up all messages for any Confluent connector to TRACE. It also returns a list of the individually loggers, from which you can cherry-pick different loggers and turn their specific loglevel up or down as required. For example:

curl -X PUT http://localhost:8083/admin/loggers/io.confluent.connect.jdbc.dialect.DatabaseDialects \
     -H "Content-Type:application/json" -d '{"level": "INFO"}'

参考:https://rmoff.net/2020/01/16/changed-the-logging-level-for-kafka-connect-dynamically/

这篇关于带有 JdbcConnectionSource 连接器的 Kafka Connect 无法创建任务(连接器正在运行但任务不是)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！