在K8 Minikube集群之外访问Kafka Broker [英] Access kafka broker outside k8 minikube cluster

问题描述

我的Mac上的minikube k8集群上的Pod上运行着一张Landoop kafka图像.我有2种不同的服务来公开端口8081(用于模式注册表)和9092(用于代理).我已经在NodePort服务中映射了端口8081-> 30081和9092-> 30092，以便可以从群集外部访问它. 但是，当我尝试运行控制台使用者或我的使用者应用程序时，Kafka从不使用消息. 要验证代理8092端口在k8集群外部是否可访问:

I have a landoop kafka image running on a Pod on minikube k8 cluster on my mac. I have 2 different services to expose the port 8081 for schema registry and 9092 for broker. I have mapped the ports 8081 -> 30081 and 9092 -> 30092 in my NodePort services so that I can access it from outside the cluster. But when I try to run a console consumer or my consumer app, Kafka never consumes messages. To verify broker 9092 port is reachable outside k8 cluster:

nc <exposed-ip> 30092, it says the port is open.

要验证架构注册表8081是否可访问:

To verify Schema registry 8081 is reachable:

curl -X GET http://192.168.99.100:30081/subjects

它返回可用的模式.

我有几个问题. 1)我们不能在k8集群之外以上述方式访问kf集群之外的Kafka吗? 2)如果端口是开放的，这并不意味着代理可用吗?

I had a couple of questions. 1) Can we not access Kafka out of k8 cluster in an above-mentioned way outside of k8 cluster?If so am I doing it wrong in some way? 2) If the port is open, doesn't that mean that broker is available?

我们将不胜感激.谢谢

推荐答案

如果您不能直接从外部路由到Pod，则从容器网络外部访问Kafka集群非常复杂.

Accessing a Kafka cluster from outside a container network is rather complicated if you cannot route directly from the outside to the pod.

首次连接到Kafka群集时，您将连接到单个代理，并且该代理返回Kafka群集内所有代理和分区的列表.然后，Kafka客户端使用该列表与特定主题所在的代理进行交互.

When you first connect to a Kafka cluster you connect to a single broker and the broker returns the list of all brokers and partitions inside the Kafka cluster. The Kafka client then uses the list to interact with the brokers where the specific topic lays.

问题是默认情况下，代理列表包含Kafka代理的内部IP.在您的情况下，这将是容器网络ip.您可以通过在每个代理的配置中设置advertised.listeners来覆盖此值.

The problem is that the broker lists contains by default the internal IP of the Kafka broker. Which would be in your case the container network ip. You can overwrite this value by setting advertised.listeners inside each broker's configuration.

要从Kubernetes外部使Kafka群集可用，您需要为每个代理配置一个节点端口服务，并将每个代理的advertised.listeners设置设置为相应节点端口服务的外部ip.但是请注意，当您尝试从Kubernetes集群内部使用Kafka时，这会增加额外的延迟和故障点.

To make a Kafka cluster available from outside Kubernetes you need to configure a nodeport service per each of your brokers and set the advertised.listeners setting of each broker to the external ip of the corresponding nodeport service. But note that this adds additional latency and failure points when you try to use Kafka from inside your Kubernetes cluster.

这篇关于在K8 Minikube集群之外访问Kafka Broker的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！