Kafka Connect 和 Kafka Broker 版本兼容性 [英] Kafka Connect and Kafka Broker version compatibility

问题描述

我们有一个运行代理版本 Apache Kafka 0.11 的Kerberized Kafka 集群".该集群由不同的团队管理，我们对此没有任何控制权.

We have a "Kerberized Kafka cluster" running brokers version Apache Kafka 0.11.This cluster is managed by a different team and we dont have any control over this.

我们现在正在尝试在我们自己的 K8S 集群上安装 Kafka Connect 集群.

We are now trying to install Kafka Connect cluster on our own K8S cluster.

我们遵循这个兼容性矩阵 https://docs.confluent.io/current/installation/versions-interoperability.html

We were following this compatibility matrix https://docs.confluent.io/current/installation/versions-interoperability.html

因此，由于 Brokers 版本为 0.11，我们不得不为 Schema Registry 和 Kafka Connect pod 坚持使用 Confluent Platform 3.3.3 映像

According to this, we had to stick with Confluent Platform 3.3.3 images for Schema Registry and Kafka Connect pods since the Brokers version is 0.11

我们还希望启用在此处实现的 Connect JMX 指标:https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework

Also we want to enable Connect JMX metrics which was implemented here : https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework

根据文档，Kafka 1.0.0 支持这些 JMX 指标

According to the documentation these JMX metrics are supported in Kafka 1.0.0

问题是:

1) 我们可以在 Kafka Brokers 0.11 版中使用最新的 Confluent 镜像(例如:5.2.0 版)吗?

1) Can we use latest Confluent images (eg: version 5.2.0) with Kafka Brokers version 0.11?

2) 使用当前设置 (Confluent 3.3.3) 我启用了所有 JMX 指标，但我们没有看到任何基于 Kafka 连接器的指标.我看到了一般的 Kafka Connect 指标，但没有看到每个连接器的指标(例如 Source 和 Sink 指标).这是因为我们使用的是旧 CP 版本吗?

2) With current setup (Confluent 3.3.3) I have enabled all the JMX metrics but we don't see any Kafka Connector based metrics. I see general Kafka Connect metrics, but not metrics per each Connector(eg. Source and Sink metrics).Is this because of the old CP version we use ?

推荐答案

Confluent Platform 3.2 及更高版本中包含的 Kafka Connect Workers 与 Confluent Platform 3.0 及更高版本中包含的任何 Kafka broker 兼容

Kafka Connect Workers that are included in Confluent Platform 3.2 and later are compatible with any Kafka broker that is included in Confluent Platform 3.0 and later

注意:Kafka Connect 不是 Confluent Platform 功能，而是基于普通的 Apache Kafka API 构建的.因此，使用 Kafka 1.0 和 2.0 客户端库的 Connect 实例仍然可以与 Kafka 0.11 代理一起使用以实现基本通信协议

Note: Kafka Connect is not a Confluent Platform feature and is built over plain Apache Kafka API. Therefore, Connect instances using Kafka 1.0 and 2.0 client libraries still work with Kafka 0.11 brokers for basic communication protocols

如果您遇到 Kerberos 问题，这些问题很可能是无关的

If you're having Kerberos issues, those are likely unrelated

我看到了一般的 Kafka Connect 指标，但没有看到每个连接器的指标

I see general Kafka Connect metrics, but not metrics per each Connector

一般来说，没有这样的每个连接器指标，例如读取/写入数据库的记录数"，或者至少，这在很大程度上取决于连接器是否实际实现了自己的指标，但是是的，3.3.x 版本均基于 Kafka 0.11，而不是 1.0，其中添加了更多 Connect API 指标

In general, there is no such per connector metric such as "number of records read/written to a database", or at least, it would largely depend if the connectors actual implement their own metrics, but yes, the 3.3.x releases are all based on Kafka 0.11, not 1.0, where more Connect API metrics were added

这篇关于Kafka Connect 和 Kafka Broker 版本兼容性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！