Kafka-MongoDB Debezium 连接器:分布式模式 [英] Kafka-MongoDB Debezium Connector : distributed mode

问题描述

我正在研究 debezium mongodb 源连接器.我可以通过将 kafka 引导服务器地址作为远程机器(部署在 Kubernetes 中)和远程 MongoDB url 来以分布式模式在本地机器上运行连接器吗?

I am working on debezium mongodb source connector. Can I run connector in local machine in distributed mode by giving kafka bootstrap server address as remote machine (deployed in Kubernetes) and remote MongoDB url?

我试过了，我看到连接器成功启动，没有错误，只有很少的警告，但没有数据从 mongodb 流出.

I tried this and I see connector starts successfully, no errors, just few warnings but no data is flowing from mongodb.

使用以下命令运行连接器

Using below command to run connector

./bin/connect-distributed ./etc/schema-registry/connect-avro-distributed.properties ./etc/kafka/connect-mongodb-source.properties

如果不是，我还能如何实现这一点，我不想像大多数教程建议的那样安装本地 kafka 或 mondoDB.我想为此使用我们的测试服务器.

If not how else can I achieve this, I donot want to install local kafka or mondoDB as most of the tutorial suggest. I want to use our test servers for this.

按照以下教程进行操作:https://medium.com/tech-that-works/cloud-kafka-connector-for-mongodb-source-8b525b779772

以下是该问题的更多详细信息连接器工作正常，我在连接器日志末尾看到以下几行

Below are more details for the issue Connector works fine, I see below lines at the end of connector log

 INFO [Worker clientId=connect-1, groupId=connect-cluster] Starting connectors and tasks using config offset -1 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1000)
] INFO [Worker clientId=connect-1, groupId=connect-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1021)

我还在/etc/kafka/connect-mongodb-source.properties 中定义了 MongoDB 配置，如下

I have also defined MongoDB config in /etc/kafka/connect-mongodb-source.properties as follows

name=mongodb-source-connector 
connector.class=io.debezium.connector.mongodb.MongoDbConnector 
mongodb.hosts=/remoteserveraddress:27017 
mongodb.name=mongo_conn 
initial.sync.max.threads=1 
tasks.max=1

但是数据不在 MongoDB 和 Kafka 之间流动.我还针对这个 Kafka-MongoDB Debezium Connector 发布了单独的问题:分布式模式

But Data is not flowing between MongoDB and Kafka. I have also posted saperate question for this Kafka-MongoDB Debezium Connector : distributed mode

任何指针都是appriciated

Any pointers are appriciated

推荐答案

connect-distributed 只接受单个属性文件.

connect-distributed only accepts a single property file.

您必须使用 REST API 在分布式模式下配置 Kafka Connect.

You must use the REST API to configure Kafka Connect in Distributed mode.

https://docs.confluent.io/current/connect/references/restapi.html

注意:默认情况下，消费者将读取主题外的最新数据，而不是现有数据.

Note: by default, the consumer will read the latest data off the topic, not existing data.

您可以将其添加到 connect-avro-distributed.properties 以修复它

You would add this to the connect-avro-distributed.properties to fix it

consumer.auto.offset.reset=earliest

这篇关于Kafka-MongoDB Debezium 连接器:分布式模式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！