在 k8 minikube 集群外访问 kafka broker [英] Access kafka broker outside k8 minikube cluster

问题描述

我在 Mac 上的 minikube k8 集群上的 Pod 上运行了一个 Landoop kafka 映像.我有 2 个不同的服务来公开模式注册表的端口 8081 和代理的 9092 端口.我已经在 NodePort 服务中映射了端口 8081 -> 30081 和 9092 -> 30092，以便我可以从集群外部访问它.但是当我尝试运行控制台消费者或我的消费者应用程序时，Kafka 从不消费消息.要验证代理 9092 端口是否可以在 k8 集群外访问:

I have a landoop kafka image running on a Pod on minikube k8 cluster on my mac. I have 2 different services to expose the port 8081 for schema registry and 9092 for broker. I have mapped the ports 8081 -> 30081 and 9092 -> 30092 in my NodePort services so that I can access it from outside the cluster. But when I try to run a console consumer or my consumer app, Kafka never consumes messages. To verify broker 9092 port is reachable outside k8 cluster:

nc <exposed-ip> 30092, it says the port is open.

要验证架构注册表 8081 是否可访问:

To verify Schema registry 8081 is reachable:

curl -X GET http://192.168.99.100:30081/subjects

它返回可用的模式.

我有几个问题.1)我们不能在k8集群之外以上述方式访问k8集群外的Kafka吗?如果是这样，我在某种程度上做错了吗?2)如果端口是开放的，是不是就意味着broker可用?

I had a couple of questions. 1) Can we not access Kafka out of k8 cluster in an above-mentioned way outside of k8 cluster?If so am I doing it wrong in some way? 2) If the port is open, doesn't that mean that broker is available?

感谢任何帮助.谢谢

推荐答案

如果不能从容器网络外部直接路由到 Pod，那么从容器网络外部访问 Kafka 集群会相当复杂.

Accessing a Kafka cluster from outside a container network is rather complicated if you cannot route directly from the outside to the pod.

当您第一次连接到 Kafka 集群时，您连接到单个代理，代理返回 Kafka 集群内所有代理和分区的列表.Kafka 客户端然后使用该列表与特定主题所在的代理进行交互.

When you first connect to a Kafka cluster you connect to a single broker and the broker returns the list of all brokers and partitions inside the Kafka cluster. The Kafka client then uses the list to interact with the brokers where the specific topic lays.

问题是代理列表默认包含 Kafka 代理的内部 IP.在您的情况下，这将是容器网络 ip.您可以通过在每个代理的配置中设置 advertised.listeners 来覆盖此值.

The problem is that the broker lists contains by default the internal IP of the Kafka broker. Which would be in your case the container network ip. You can overwrite this value by setting advertised.listeners inside each broker's configuration.

要使 Kafka 集群从 Kubernetes 外部可用，您需要为每个代理配置一个 nodeport 服务，并将每个代理的 advertised.listeners 设置设置为相应 nodeport 服务的外部 ip.但请注意，当您尝试从 Kubernetes 集群内部使用 Kafka 时，这会增加额外的延迟和故障点.

To make a Kafka cluster available from outside Kubernetes you need to configure a nodeport service per each of your brokers and set the advertised.listeners setting of each broker to the external ip of the corresponding nodeport service. But note that this adds additional latency and failure points when you try to use Kafka from inside your Kubernetes cluster.

这篇关于在 k8 minikube 集群外访问 kafka broker的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！