kafka 是否可以连接 - mongo 源作为集群运行(max.tasks > 1) [英] Can kafka connect - mongo source run as cluster (max.tasks > 1)
问题描述
我正在使用以下 mongo-source由 kafka-connect 支持.我发现了 mongo 源的配置之一(来自 here) 是 tasks.max.
I'm using the following mongo-source which is supported by kafka-connect. I found that one of the configurations of the mongo source (from here) is tasks.max.
这意味着我可以提供大于 1 的连接器 tasks.max，但我不明白它在幕后会做什么?
this means I can provide the connector tasks.max which is > 1, but I fail to understand what it will do behind the scene?
如果它会创建多个连接器来监听 mongoDb 更改流，那么我最终会得到重复的消息.那么，mongo-source 真的具有并行性并且可以作为集群工作吗?如果它有超过 1 个 task.max，它会做什么?
If it will create multiple connectors to listen to mongoDb change stream, then I will end up with duplicate messages. So, does mongo-source really has parallelism and works as a cluster? what does it do if it has more then 1 tasks.max?
推荐答案
Mongo-source 不支持 tasks.max > 1.即使您将其设置为大于 1，也只有 一个 任务会将数据从 mongo 拉到 Kafka.
Mongo-source doesn't support tasks.max > 1. Even if you set it greater than 1 only one task will be pulling data from mongo to Kafka.
创建多少任务取决于特定的连接器.函数List，(应该在连接器的实现过程中被覆盖)返回列表，该列表的大小决定了任务的数量.如果您检查 mongo-kafka 源连接器，您会看到它是 singletonList.
How many task is created depends on particular connector. Function List<Map<String, String>> Connector::taskConfigs(int maxTasks), (that should be overridden during the implementation of your connector) return the list, which size determine number of Tasks.
If you check mongo-kafka source connector you will see, that it is singletonList.
这篇关于kafka 是否可以连接 - mongo 源作为集群运行(max.tasks > 1)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
