使用JdbcConnectionSource连接器的Kafka Connect无法创建任务(连接器正在运行，但任务不在) [英] Kafka Connect with a JdbcConnectionSource connector fails to create task (connector is RUNNING but task is not)

问题描述

似乎我经常根据查询从JdbcConnectionSource创建一个Kafka Connect连接器，并成功创建状态为"RUNNING"的连接器，但未创建任何任务.在我的容器的控制台日志中，我看不到任何提示可以告诉我的错误:没有错误，没有警告，没有说明任务失败的原因.我可以使其他连接器正常工作，但有时不能.

It seems like rather often I create a Kafka Connect connector from the JdbcConnectionSource based on a query, and the connector is created successsfully with status "RUNNING", but no task is created. Looking in the console logs of my container, I see no indication that anything is wrong that I can tell: no errors, no warnings, no explanation of why the task failed. I can get other connectors to work, but sometimes one doesn't.

当连接器无法创建RUNNING任务时，如何获取更多信息以进行故障排除?

我将在下面发布我的连接器配置示例.

I'll post an example of my connector config below.

我正在使用Kafka Connect 5.4.1-ccs.

I am using Kafka Connect 5.4.1-ccs.

连接器配置(它是JDBC背后的Oracle数据库):

Connector config (it is an Oracle Database behind JDBC):

{
    "name": "FiscalYear",
    "config": {
        "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
        "tasks.max": 1,
        "connection.url": "jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=myhost.example.com)(PORT=1521))(LOAD_BALANCE=OFF)(FAILOVER=OFF)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=MY_DB_PRI)(UR=A)))",
        "connection.user":"myuser",
        "connection.password":"mypass",
        "mode": "timestamp",
        "timestamp.column.name": "MAINT_TS",
        "topic.prefix": "MyTeam.MyTopicName",
        "poll.interval.ms": 5000,
        "value.converter" : "org.apache.kafka.connect.json.JsonConverter",
        "value.converter.schemas.enable": "false",
        "numeric.mapping": "best_fit",

        "_comment": "The query is wrapped in `select * from ()` so that JdbcSourceConnector can automatically append a WHERE clause.",
        "query": "SELECT * FROM (SELECT fy_nbr, min(fy_strt_dt) fy_strt_dt, max(fy_end_dt) fy_end_dt FROM myuser.fsc_dt fd WHERE fd.fy_nbr >= 2020 and fd.fy_nbr < 2022 group by fy_nbr)/* outer query must have no WHERE clause so that the source connector can append one of its own */"
    }
}  

还有创建我的工作程序的Dockerfile:

And the Dockerfile that creates my worker:

FROM confluentinc/cp-kafka-connect:latest

# each "CONNECT_" env var refers to a Kafka Connect setting; e.g. CONNECT_REST_PORT refers to setting rest.port
#  see also https://docs.confluent.io/current/connect/references/allconfigs.html

ENV CONNECT_BOOTSTRAP_SERVERS="d.mybroker.example.com:9092"
ENV CONNECT_REST_PORT="8083"
ENV CONNECT_GROUP_ID="MyGroup2" 

ENV CONNECT_CONFIG_STORAGE_TOPIC="MyTeam.ConnectorConfig" 
ENV CONNECT_OFFSET_STORAGE_TOPIC="MyTeam.ConnectorOffsets" 
ENV CONNECT_STATUS_STORAGE_TOPIC="MyTeam.ConnectorStatus" 

ENV CONNECT_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" 
ENV CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" 

ENV CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter"  
ENV CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" 

ENV CONNECT_LOG4J_ROOT_LOGLEVEL="INFO"

COPY ojdbcDrivers /usr/share/java/kafka-connect-jdbc

(我还通过Helm图表设置了REST通告的主机名环境变量，因此这就是上面未设置的原因.)

(I also set the REST advertised hostname environment variable via my Helm chart, so that's why it isn't set in the above.)

旋转起来后，我创建了连接器，然后从REST"/status"获得它:

After it spins up, I create the connector, then get this from the REST "/status":

{"name":"FiscalYear","connector":{"state":"RUNNING","worker_id":"10.1.2.3:8083"},"tasks":[],"type":"source"}

推荐答案

当连接器无法创建RUNNING任务时，如何获取更多信息以进行故障排除?

How can one get more information to troubleshoot when a connector fails to create a RUNNING task?

我会增加您的Kafka Connect工作者的日志记录级别. 由于您使用的是Apache Kafka 2.4，因此可以动态地执行此操作，这非常有用.向您的Kafka Connect工作者发出此REST API调用:

I would increase the logging level on your Kafka Connect worker. Since you're using Apache Kafka 2.4 you can do this dynamically, which is rather useful. Issue this REST API call to your Kafka Connect worker:

curl -X PUT http://localhost:8083/admin/loggers/io.confluent \
     -H "Content-Type:application/json" -d '{"level": "TRACE"}'

这会将所有Confluent连接器的所有消息都弹出到TRACE.它还会返回各个记录器的列表，您可以从中选择不同的记录器，并根据需要将其特定的日志级别调高或调低.例如:

This bumps up all messages for any Confluent connector to TRACE. It also returns a list of the individually loggers, from which you can cherry-pick different loggers and turn their specific loglevel up or down as required. For example:

curl -X PUT http://localhost:8083/admin/loggers/io.confluent.connect.jdbc.dialect.DatabaseDialects \
     -H "Content-Type:application/json" -d '{"level": "INFO"}'

引用: https://rmoff.net/2020/01/16/动态更改kafka-connecting-the-logging-level-/

这篇关于使用JdbcConnectionSource连接器的Kafka Connect无法创建任务(连接器正在运行，但任务不在)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！