无法使用 SSL 在 Kafka 中找出 inter.broker.listener.name 的设置 [英] Can't figure out setting for inter.broker.listener.name in Kafka with SSL

问题描述

我正在尝试使用 SSL (TLS) 节点间以及节点和客户端之间配置 Kafka 节点，但遇到了配置问题.卡夫卡版本是 2.3.0.我的相关设置是:

I am attempting to configure Kafka nodes with SSL (TLS) inter-nodes and between nodes and clients but run into configuration problems. Kafka version is 2.3.0. My relevant settings are:

      - KAFKA_BROKER_ID=1
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_LISTENERS=LISTENER_INTERNAL://kafka1:9092,LISTENER_EXTERNAL://kafka1:29092
      - KAFKA_ADVERTISED_LISTENERS=LISTENER_INTERNAL://kafka1:9092,LISTENER_EXTERNAL://localhost:29091
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=LISTENER_INTERNAL:SSL,LISTENER_EXTERNAL:SSL
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181
      - KAFKA_AUTO_CREATE_TOPICS_ENABLE=false
      - KAFKA_SSL_TRUSTSTORE_LOCATION=/var/private/ssl/server.truststore.jks
      - KAFKA_SSL_TRUSTSTORE_PASSWORD=changeit
      - KAFKA_SSL_KEYSTORE_LOCATION=/var/private/ssl/server.keystore.jks
      - KAFKA_SSL_KEYSTORE_PASSWORD=changeit
      - KAFKA_SSL_KEY_PASSWORD=changeit
      - KAFKA_SECURITY_INTER_BROKER_PROTOCOL=SSL
      - KAFKA_SSL_CLIENT_AUTH=required

仅供参考，为简单起见，我从实例化 Kafka 容器的 docker-compose 文件中复制了设置.环境变量 1:1 映射到 server.properties 中的属性.在容器启动期间，这些设置会应用于 server.properties 文件.

FYI, for simplicity I copied the settings from the docker-compose file that instantiates the Kafka container. The env vars map 1:1 to properties in server.properties. During container start, these settings are applied to the server.properties file.

当我开始使用此配置时，收到以下错误消息:

When I start with this configuration, I receive the following error message:

java.lang.IllegalArgumentException:要求失败:inter.broker.listener.name 必须是advertised.listeners 中定义的侦听器名称.基于当前配置的监听器的有效选项是 LISTENER_INTERNAL,LISTENER_EXTERNAL

java.lang.IllegalArgumentException: requirement failed: inter.broker.listener.name must be a listener name defined in advertised.listeners. The valid options based on currently configured listeners are LISTENER_INTERNAL,LISTENER_EXTERNAL

当我将 inter.broker.listener.name 属性设置为 INTERNAL_LISTENER、SSL、null 时或空字符串，我收到此错误消息:

When I set the inter.broker.listener.name property to either INTERNAL_LISTENER, SSL, null or empty string, I receive instead this error message:

org.apache.kafka.common.config.ConfigException: 只应设置 inter.broker.listener.name 和 security.inter.broker.protocol 之一.

org.apache.kafka.common.config.ConfigException: Only one of inter.broker.listener.name and security.inter.broker.protocol should be set.

我在这个问题上花了几个小时.我已经将我的设置与网上那些应该用 SSL 配置演示 Kafka 的几个例子进行了比较.

I have spent a few hours on this issue. I have compared my settings to those few examples on the web that are supposed to demonstrate Kafka with SSL configuration.

有什么想法吗?

推荐答案

我终于想出了如何拥有多个 SSL 侦听器.我会在这里记录这个以防其他人遇到同样的问题，因为多个 SSL 侦听器的工作示例似乎很少甚至不存在.以下是我的工作配置(仅显示从 docker-compose 传递过来的相关属性):

I finally figured out how to have multiple SSL listeners. I'll document this here in case someone else runs into the same issue, since working examples of multiple SSL listeners seem to be rare to non-existent. Below is my working configuration (only showing the relevant properties passed through from docker-compose):

ALLOW_PLAINTEXT_LISTENER=no
KAFKA_LISTENERS=ISSL://kafka1:9092,OSSL://kafka1:29092
KAFKA_ADVERTISED_LISTENERS=ISSL://kafka1:9092,OSSL://localhost:29092
KAFKA_INTER_BROKER_LISTENER_NAME=ISSL
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=ISSL:SSL,OSSL:SSL
KAFKA_SSL_CLIENT_AUTH=required

关键是不要配置 KAFKA_SECURITY_INTER_BROKER_PROTOCOL，因为它与 KAFKA_INTER_BROKER_LISTENER_NAME 键是互斥的.

The key to this was to NOT configure the KAFKA_SECURITY_INTER_BROKER_PROTOCOL as it is mutually exclusive with the KAFKA_INTER_BROKER_LISTENER_NAME key.

如果有多个听众，似乎需要KAFKA_LISTENER_SECURITY_PROTOCOL_MAP和KAFKA_INTER_BROKER_LISTENER_NAME的组合.

In case of multiple listeners, it seems that the combination of KAFKA_LISTENER_SECURITY_PROTOCOL_MAP and KAFKA_INTER_BROKER_LISTENER_NAME is what is required.

这篇关于无法使用 SSL 在 Kafka 中找出 inter.broker.listener.name 的设置的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！