Kafka docker 撰写外部连接 [英] Kafka docker compose external connection

问题描述

我想将 9093 发布到 docker 容器之外.当我将 kafka-0 端口设置为 9093 和 KAFKA_ADVERTISED_LISTENERS 如下时，我无法连接到 localhost:9093，如下面的 docker-compose 文件所示.

I want to expost 9093 to outside of the docker container. When I set the kafka-0 ports exposed to 9093 and the KAFKA_ADVERTISED_LISTENERS as follow, i am unable to connect to localhost:9093 as shown in the following docker-compose file.

version: '3'

services:
  kafka-0:
    image: confluentinc/cp-kafka:5.2.1
    container_name: kafka-0
    hostname: kafka-0
    ports:
      - "9093:9092"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=wise-nlp-zookeeper:2181
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
    depends_on:
      - zookeeper

  zookeeper:
    image: confluentinc/cp-zookeeper:5.3.1
    container_name: zookeeper
    ports:
      - "2182:2181"
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181

但是，当我更改为

ports:
 - "9092:9092"

和

- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9092

我能够连接到 kafka 代理 localhost:9092.

I am able to connect to kafka broker localhost:9092.

如何将外部端口更改为 9093 以便应用程序连接?我想设置多个经纪人.

How can i change external port to 9093 for applications to connect? I want to set up multiple brokers.

推荐答案

为什么它目前不起作用?

通告的侦听器(在KAFKA_ADVERTISED_LISTENERS中定义)是代理在其初始连接中返回给客户端以供其在后续连接中使用的主机和端口.

Why's it not working currently?

Advertised listener(s) (as defined in KAFKA_ADVERTISED_LISTENERS) are the host and port that the broker returns to the client in its initial connection for it to use in subsequent connections.

如果您希望外部客户端使用 9093，则 KAFKA_ADVERTISED_LISTENERS=…PLAINTEXT_HOST://localhost:9093 是正确的.但是，您还没有配置您的 KAFKA_LISTENERS，如果您在启动时检查代理日志，它将默认为 KAFKA_ADVERTISED_LISTENERS 设置的值:

If you want external clients to use 9093 then KAFKA_ADVERTISED_LISTENERS=…PLAINTEXT_HOST://localhost:9093 is correct. However, you've not configured your KAFKA_LISTENERS, which if you check the broker log when it starts up will default to the value set by KAFKA_ADVERTISED_LISTENERS:

kafka-0      |  listeners = PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9093

因此，在这种状态下，代理正在侦听端口 9093，但是通过此 Docker Compose 指令，您已将外部连接重定向到容器内的 9093 到 9092 上的容器:

So in this state, the broker is listening on port 9093, but with this Docker Compose instruction you've redirected external connections into the container on 9093 to 9092 within the container:

    ports: 
      - "9093:9092"

➜ docker ps
CONTAINER ID        IMAGE                             … PORTS                                        NAMES
8b934ef4145c        confluentinc/cp-kafka:5.4.1       … 0.0.0.0:9093->9092/tcp                       kafka-0

因此，您的外部连接将转到容器中 的端口 9092 — 而代理并未侦听此端口.您可以使用 nc 验证这一点:

So your external connections will go to port 9092 in the container—and the broker is not listening on this port. You can verify this with nc:

-- Port 9093 is open on the host machine
➜ nc -vz localhost 9093
Connection to localhost port 9093 [tcp/*] succeeded!

-- Port 9092 is _not_ open on the Kafka container
➜ docker exec -it kafka-0 nc -vz localhost 9092
localhost [127.0.0.1] 9092 (?) : Connection refused

❌ 您会看到客户端连接失败

❌ You'll see that a client connection fails

➜ kafkacat -b localhost:9093 -L
% ERROR: Failed to acquire metadata: Local: Broker transport failure

如何解决?

您可以:

1. 将侦听器更改为您使用 Docker 端口重定向的目标端口.这会起作用，但我个人认为更令人困惑.
2. 更改 Docker 端口重定向以定位侦听器所在的端口.这是我会使用的选项，因为它更清晰(例如，始终使用端口 9093，而不是将 9092 和 9093 混合在一起)

1. Change the listener to be on the port that you target with the Docker port redirect. This will work but personally I think is more confusing.
2. Change the Docker port redirect to target the port on which the listener is on. This is the option I would use as it is clearer (e.g. port 9093 is used throughout, rather than mixing 9092 and 9093 together)

选项 1:将侦听器更改为使用 Docker 端口重定向的目标端口

version: '3'

services:
  kafka-0:
    image: confluentinc/cp-kafka:5.4.1
    container_name: kafka-0
    ports:
      - "9093:9092"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
      - KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9092
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
    depends_on:
      - zookeeper

  zookeeper:
    image: confluentinc/cp-zookeeper:5.4.1
    container_name: zookeeper
    ports:
      - "2182:2181"
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181

✅测试:

➜ kafkacat -b localhost:9093 -L
Metadata for all topics (from broker 1: localhost:9093/1):
 1 brokers:
  broker 1 at localhost:9093 (controller)

选项 2:更改 Docker 端口重定向以定位侦听器所在的端口

version: '3'

services:
  kafka-0:
    image: confluentinc/cp-kafka:5.4.1
    container_name: kafka-0
    ports:
      - "9093:9093"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka-0:29094,PLAINTEXT_HOST://localhost:9093
      # If you don't specify KAFKA_LISTENERS it will default to the ports used in
      # KAFKA_ADVERTISED_LISTENERS, but IMO it's better to be explicit about these settings
      - KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:29094,PLAINTEXT_HOST://0.0.0.0:9093
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
    depends_on:
      - zookeeper

  zookeeper:
    image: confluentinc/cp-zookeeper:5.4.1
    container_name: zookeeper
    ports:
      - "2182:2181"
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181

✅测试

➜ kafkacat -b localhost:9093 -L
Metadata for all topics (from broker 1: localhost:9093/1):
 1 brokers:
  broker 1 at localhost:9093 (controller)

从 Docker 网络内连接到 Kafka

以上示例是关于从 Docker 主机连接到 Kafka.如果您想从 Docker 网络内(例如另一个容器)连接到它，您需要使用 kafka-0:29094 作为代理主机和 IP.如果您尝试使用 localhost:9093，那么客户端容器会将 localhost 解析为它自己的容器，从而失败.

Connecting to Kafka from within the Docker network

The examples above are about connecting to Kafka from the Docker host. If you want to connect to it from within the Docker network (e.g. another container) you need to use kafka-0:29094 as the broker host and IP. If you try to use localhost:9093 then the client container will resolve localhost to its own container, and thus fail.

见这里 例如 Docker Compose 与多个 Kafka 代理.

See here for an example Docker Compose with multiple Kafka brokers.

https://rmoff.net/2018/08/02/kafka-listeners-explained/

这篇关于Kafka docker 撰写外部连接的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！