从本地计算机连接到Docker中运行的Kafka [英] Connect to Kafka running in Docker from local machine

问题描述

我在本地计算机上使用docker设置单节点基本Kafka部署，如 Confluent Kafka文档（步骤2-3）。

I setup Single Node Basic Kafka Deployment using docker on my local machine like it is described in the Confluent Kafka documentation (steps 2-3).

此外，我还暴露了zookeeper的端口2181和kafka的端口9092，以便我能够从本地机器上运行的java客户端连接到它们：

In addition, I also exposed zookeeper's port 2181 and kafka's port 9092 so that I'll be able to connect to them from java client running on local machine:

$ docker run -d \
    -p 2181:2181 \
    --net=confluent \
    --name=zookeeper \
    -e ZOOKEEPER_CLIENT_PORT=2181 \
    confluentinc/cp-zookeeper:4.1.0

$ docker run -d \
    --net=confluent \
    --name=kafka \
    -p 9092:9092 \
    -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    confluentinc/cp-kafka:4.1.0

问题：当我尝试从主机连接到kafka时，连接失败，因为它无法解析地址：kafka：9092。

Problem: When I try to connect to kafka from the host machine, the connection fails because it can't resolve address: kafka:9092.

这是我的Java代码：

Here is my Java code:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("client.id", "KafkaExampleProducer");
props.put("key.serializer", LongSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
KafkaProducer<Long, String> producer = new KafkaProducer<>(props);
ProducerRecord<Long, String> record = new ProducerRecord<>("foo", 1L, "Test 1");
producer.send(record).get();
producer.flush();

例外：

java.io.IOException: Can't resolve address: kafka:9092
    at org.apache.kafka.common.network.Selector.doConnect(Selector.java:235) ~[kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.common.network.Selector.connect(Selector.java:214) ~[kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:864) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:265) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:266) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:176) [kafka-clients-2.0.0.jar:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.nio.channels.UnresolvedAddressException: null
    at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_144]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_144]
    at org.apache.kafka.common.network.Selector.doConnect(Selector.java:233) ~[kafka-clients-2.0.0.jar:na]
    ... 7 common frames omitted

问题：如何从主机连接到Docker中运行的kafka？

Question: How to connect to kafka running in Docker from host machine?

注意：我知道我理论上可以玩使用DNS设置和 / etc / hosts 但这是一种解决方法 - 它应该不是那样的。

Note: I know that I could theoretically play around with DNS setup and /etc/hosts but it is a workaround - it shouldn't be like that.

此处也存在类似问题，但它基于 ches / kafka 图像。我使用基于汇合的图片，这是不一样的。

There is also similar question here, however it is based on ches/kafka image. I use confluent based image which is not the same.

推荐答案

_{免责声明：以下使用 confluentinc docker images，而不是 wurstmeister / kafa ，虽然有类似的配置，但我还没试过。同时 Docker Compose中的这一行。}

_{Disclaimer: The following uses confluentinc docker images, not wurstmeister/kafa, although there is a similar configuration, I have not tried it. Also this line in the Docker Compose.}

Confluent快速入门文档假设所有生产和消费请求都在Docker网络中。

That Confluent quickstart document assumes all produce and consume requests will be within the Docker network.

您可以通过在自己的容器中运行Kafka客户端代码来解决问题，但是否则您需要添加一些环境变量以在外部公开容器，同时仍然在Docker网络中工作。

You could fix the problem by running your Kafka client code within its own container, but otherwise you'll need to add some more environment variables for exposing the container externally, while still having it work within the Docker network.

首先添加 PLAINTEXT_HOST：PLAINTEXT 的协议映射，它将侦听器协议映射到Kafka协议

First add a protocol mapping of PLAINTEXT_HOST:PLAINTEXT that will map the listener protocol to a Kafka protocol

-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP = PLAINTEXT：PLAINTEXT，PLAINTEXT_HOST：PLAINTEXT

然后在不同的端口上设置两个广告的侦听器。 （ kafka：9092 这里指的是docker容器名称）。请注意，协议匹配上面映射的右侧值

Then setup two advertised listeners on different ports. (kafka:9092 here refers to the docker container name). Notice the protocols match the right side values of the mappings above

-e KAFKA_ADVERTISED_LISTENERS = PLAINTEXT：// kafka：9092，PLAINTEXT_HOST：// localhost ：29092

运行容器时，添加 -p 29092：29092 主机端口映射

When running the container, add -p 29092:29092 for the host port mapping

当运行任何Kafka客户端外部 Docker网络时，请使用 localhost：29092

When running any Kafka Client outside the Docker network, use localhost:29092

在Docker网络中运行应用程序 时，请使用 kafka：9092

When running an app in the Docker network, use kafka:9092

请参阅示例撰写完整Confluent堆栈的文件

这篇关于从本地计算机连接到Docker中运行的Kafka的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！