连接到在Docker中运行的Kafka [英] Connect to Kafka running in Docker

问题描述

我在本地计算机上设置了一个单节点Kafka Docker容器，如 Confluent文档（第2-3步）。

I setup a single node Kafka Docker container on my local machine like it is described in the Confluent documentation (steps 2-3).

此外，我还公开了Zookeeper的端口2181和Kafka的端口9092，这样我就可以从在本地计算机上运行的客户端连接到它们：

In addition, I also exposed Zookeeper's port 2181 and Kafka's port 9092 so that I'll be able to connect to them from a client running on local machine:

$ docker run -d \
    -p 2181:2181 \
    --net=confluent \
    --name=zookeeper \
    -e ZOOKEEPER_CLIENT_PORT=2181 \
    confluentinc/cp-zookeeper:4.1.0

$ docker run -d \
    --net=confluent \
    --name=kafka \
    -p 9092:9092 \
    -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    confluentinc/cp-kafka:4.1.0

问题：：当我尝试从主机连接到Kafka时，连接失败，因为它无法解析地址：kafka：9092 。

Problem: When I try to connect to Kafka from the host machine, the connection fails because it can't resolve address: kafka:9092.

这是我的Java代码：

Here is my Java code:

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("client.id", "KafkaExampleProducer");
props.put("key.serializer", LongSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
KafkaProducer<Long, String> producer = new KafkaProducer<>(props);
ProducerRecord<Long, String> record = new ProducerRecord<>("foo", 1L, "Test 1");
producer.send(record).get();
producer.flush();

例外：

java.io.IOException: Can't resolve address: kafka:9092
    at org.apache.kafka.common.network.Selector.doConnect(Selector.java:235) ~[kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.common.network.Selector.connect(Selector.java:214) ~[kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:864) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:265) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.sendProducerData(Sender.java:266) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238) [kafka-clients-2.0.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:176) [kafka-clients-2.0.0.jar:na]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.nio.channels.UnresolvedAddressException: null
    at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_144]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_144]
    at org.apache.kafka.common.network.Selector.doConnect(Selector.java:233) ~[kafka-clients-2.0.0.jar:na]
    ... 7 common frames omitted

问题：：如何连接到在Docker中运行的Kafka？我的代码是从主机而不是Docker运行的。

Question: How to connect to Kafka running in Docker? My code is running from host machine, not Docker.

注意：我知道从理论上讲我可以使用DNS设置和 / etc / hosts 解决方法-不应该那样。

Note: I know that I could theoretically play around with DNS setup and /etc/hosts but it is a workaround - it shouldn't be like that.

还有类似的问题 ，但这是基于 ches / kafka 图片的。我使用基于 confluentinc 的图像，这是不一样的。

There is also similar question here, however it is based on ches/kafka image. I use confluentinc based image which is not the same.

推荐答案

免责声明

Disclaimer

tl; dr -在一天结束时，它们都是相同的在容器中运行的Apache Kafka 。您只依赖于其配置方式。

tl;dr - At the end of the day, it's all the same Apache Kafka running in a container. You're just dependent on how it is configured. And which variables make it so.

_{以下使用 confluentinc docker images ，而不是 wurstmeister / kafka ，尽管也有类似的配置，但我没有尝试过。如果使用该图像，请阅读其连接性维基。}

_{The following uses confluentinc docker images, not wurstmeister/kafka, although there is a similar configuration, I have not tried it. If using that image, read their Connectivity wiki.}

_{与 wurstmeister 图像完全相同，但是它是社区维护的，而不是内置在自动CI / CD发行版中的。同样具有简约性，并且可以在多个云提供商中运行}。 _{对于 bitnami Kafka图像，请参阅自述文件}

_{Nothing against the wurstmeister image, but it's community maintained, not built in an automated CI/CD release... Bitnami ones are similarly minimalistic and run in multiple cloud providers}. _{For bitnami Kafka images, refer their README}

_{debezium / kafka 上的文档在这里提到。
注意：不推荐使用公告的主机和端口设置。广告监听器同时涵盖了}

_{spotify / kafka 已被弃用和过时。 fast-data-dev 对于一个完整的解决方案非常有用，但是它很肿}

_{spotify/kafka is deprecated and outdated. fast-data-dev is great for an all in one solution, but it is bloated}

有关补充阅读，功能齐全 docker-compose 和网络图的信息，请参见此博客作者@rmoff

For supplemental reading, a fully-functional docker-compose, and network diagrams, see this blog by @rmoff

< h1>答案

Confluent快速入门（Docker）文档假定所有生产和消费请求都将在Docker网络内。

Answer

The Confluent quickstart (Docker) document assumes all produce and consume requests will be within the Docker network.

您可以通过在自己的容器中运行Kafka客户端代码来解决此问题，但否则，您需要添加更多环境变量以在外部暴露容器，而仍然可以在Docker网络中运行。

You could fix the problem by running your Kafka client code within its own container, but otherwise you'll need to add some more environment variables for exposing the container externally, while still having it work within the Docker network.

首先添加 PLAINTEXT_HOST：PLAINTEXT 的协议映射，它将侦听器协议映射到Kafka协议

First add a protocol mapping of PLAINTEXT_HOST:PLAINTEXT that will map the listener protocol to a Kafka protocol

-e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP = PLAINTEXT：PLAINTEXT，PLAINTEXT_HOST：PLAINTEXT

然后在不同的端口上设置两个播发的侦听器。 （ kafka：9092 这里指的是Docker容器名称）。请注意，协议与上面的映射的右侧值匹配

Then setup two advertised listeners on different ports. (kafka:9092 here refers to the docker container name). Notice the protocols match the right side values of the mappings above

-e KAFKA_ADVERTISED_LISTENERS = PLAINTEXT：// kafka：9092，PLAINTEXT_HOST：// localhost ：29092

运行容器时，将 -p 29092：29092 添加到主机端口映射

When running the container, add -p 29092:29092 for the host port mapping

tl; dr （具有上述设置）

何时在外部 Docker网络上运行任何Kafka客户端（包括您可能已在本地安装的CLI工具），将 localhost：29092 用于引导服务器，将 localhost：2181 for Zookeeper

When running any Kafka Client outside the Docker network (including CLI tools you might have installed locally), use localhost:29092 for bootstrap servers and localhost:2181 for Zookeeper

在Docker网络中运行应用时，使用 kafka：9092 用于引导服务器， zookeeper：2181 用于Zookeeper

When running an app in the Docker network, use kafka:9092 for bootstrap servers and zookeeper:2181 for Zookeeper

参见示例撰写完整的融合堆栈文件

对于对 Kubernetes 部署感兴趣的任何人： https://operatorhub.io/?keyword=Kafka

For anyone interested in Kubernetes deployments: https://operatorhub.io/?keyword=Kafka

这篇关于连接到在Docker中运行的Kafka的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！