Kafka-MongoDB Debezium连接器:分布式模式 [英] Kafka-MongoDB Debezium Connector : distributed mode
问题描述
我正在使用debezium mongodb源连接器.是否可以通过将kafka引导程序服务器地址指定为远程计算机(在Kubernetes中部署)和远程MongoDB url,以分布式模式在本地计算机上运行连接器?
I am working on debezium mongodb source connector. Can I run connector in local machine in distributed mode by giving kafka bootstrap server address as remote machine (deployed in Kubernetes) and remote MongoDB url?
我尝试了一下,发现连接器成功启动,没有错误,只有很少的警告,但没有数据从mongodb流出.
I tried this and I see connector starts successfully, no errors, just few warnings but no data is flowing from mongodb.
使用以下命令运行连接器
Using below command to run connector
./bin/connect-distributed ./etc/schema-registry/connect-avro-distributed.properties ./etc/kafka/connect-mongodb-source.properties
如果没有其他解决方法,我不想按照本教程的大多数建议安装本地kafka或mondoDB.我想为此使用我们的测试服务器.
If not how else can I achieve this, I donot want to install local kafka or mondoDB as most of the tutorial suggest. I want to use our test servers for this.
以下是针对此的教程 : https://medium.com/tech-that-works/cloud-kafka-connector-for-mongodb-source-8b525b779772
Followed below tutorial for this : https://medium.com/tech-that-works/cloud-kafka-connector-for-mongodb-source-8b525b779772
以下是该问题的更多详细信息 连接器工作正常,我在连接器日志末尾看到以下几行
Below are more details for the issue Connector works fine, I see below lines at the end of connector log
INFO [Worker clientId=connect-1, groupId=connect-cluster] Starting connectors and tasks using config offset -1 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1000)
] INFO [Worker clientId=connect-1, groupId=connect-cluster] Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1021)
我还在/etc/kafka/connect-mongodb-source.properties中定义了MongoDB配置,如下所示
I have also defined MongoDB config in /etc/kafka/connect-mongodb-source.properties as follows
name=mongodb-source-connector
connector.class=io.debezium.connector.mongodb.MongoDbConnector
mongodb.hosts=/remoteserveraddress:27017
mongodb.name=mongo_conn
initial.sync.max.threads=1
tasks.max=1
但是MongoDB和Kafka之间没有数据流动.我还对此Kafka-MongoDB Debezium Connector发布了一个棘手的问题:分布式模式
But Data is not flowing between MongoDB and Kafka. I have also posted saperate question for this Kafka-MongoDB Debezium Connector : distributed mode
任何指针都适用
推荐答案
connect-distributed
仅接受单个属性文件.
connect-distributed
only accepts a single property file.
您必须使用REST API在分布式模式下配置Kafka Connect.
You must use the REST API to configure Kafka Connect in Distributed mode.
https://docs.confluent.io/current/connect/references/restapi.html
注意:默认情况下,使用者将从主题中读取最新数据,而不是现有数据.
Note: by default, the consumer will read the latest data off the topic, not existing data.
您将其添加到connect-avro-distributed.properties
进行修复
consumer.auto.offset.reset=earliest
这篇关于Kafka-MongoDB Debezium连接器:分布式模式的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!