在创建和debezium源连接器时会创建哪些额外主题 [英] What are the extra topics created when creating and debezium source connector

问题描述

Q1)以下是我在为MySQL源创建Kafka连接器时使用的配置。

{
  "connector.class": "io.debezium.connector.mysql.MySqlConnector",
  "snapshot.locking.mode": "minimal",
  "database.user": "cdc_user",
  "tasks.max": "3",
  "database.history.kafka.bootstrap.servers": "10.49.115.X:9092,10.48.X.211:9092,10.X.178.121:9092,10.53.4.X:9092",
  "database.history.kafka.topic": "history.cdc.fkw.supply.mp.seller_platform",
  "database.server.name": "cdc.fkw.supply.mp",
  "heartbeat.interval.ms": "5000",
  "database.port": "3306",
  "table.whitelist": "seller_platform.Contacts, seller_platform.EmailVerificationConfigs, seller_platform.financial_account_tag, seller_platform.HolidayConfigs, seller_platform.Preferences, seller_platform.Sellers",
  "database.hostname": "something.cloud.in",
  "database.password": "ABCDE",
  "database.history.kafka.recovery.poll.interval.ms": "5000",
  "name": "cdc.fkw.supply.mp.seller_platform.connector",
  "database.history.skip.unparseable.ddl": "true",
  "errors.tolerance": "all",
  "database.whitelist": "seller_platform",
  "snapshot.mode": "when_needed"
}


curl -s --location --request GET "http://10.24.18.167:80/connectors/cdc.fkw.supply.mp.seller_platform.connector/topics" | jq '.'
{
  "cdc.fkw.supply.mp.seller_platform.connector": {
    "topics": [
      "cdc.fkw.supply.mp.seller_platform.Sellers",
      "cdc.fkw.supply.mp",
      "cdc.fkw.supply.mp.seller_platform.HolidayConfigs",
      "cdc.fkw.supply.mp.seller_platform.EmailVerificationConfigs",
      "cdc.fkw.supply.mp.seller_platform.Contacts",
      "cdc.fkw.supply.mp.seller_platform.Preferences",
      "__debezium-heartbeat.cdc.fkw.supply.mp",
      "cdc.fkw.supply.mp.seller_platform.financial_account_tag"
    ]
  }
}

为什么创建cdc.fkw.Supply.MP和__debezium-Hearbeat.cdc.fkw.Supply.MP主题？

我在这两个主题中看到一些垃圾数据。

Q2)

有没有REST API可以知道Worker服务器上的Kafka Connect转换器配置？ 如果没有API，那么我们存储所有工作属性的配置文件的路径是什么？

这是Worker属性的链接： https://docs.confluent.io/platform/current/connect/references/allconfigs.html

curl -s --location --request GET "http://10.24.18.167:80"         
                                                            
{"version":"6.1.1-ccs","commit":"c209f70c6c2e52ae","kafka_cluster_id":"snBlf-kfTdCYWEO9IIEXTA"}%

推荐答案

a1)

心跳主题存储连接器正在使用的所有Kafka主题的详细信息，以便连接器可以向其发送心跳。

创建名为Theme的lobase.server.name值以存储数据库中发生的任何架构更改。

https://debezium.io/documentation/reference/1.7/connectors/mysql.html#mysql-schema-change-topic

这篇关于在创建和debezium源连接器时会创建哪些额外主题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！