Kubernetes 集群中的 Kafka - 如何从 Kubernetes 集群外部发布/消费消息 [英] Kafka in Kubernetes Cluster- How to publish/consume messages from outside of Kubernetes Cluster

问题描述

1. 我在 Kubernetes 集群中部署并运行了 Kafka.我正在使用来自 docker hub 的这个图像 - https://hub.docker.com/r/cloudtrackinc/kubernetes-kafka/
2. 我的 kubernetes 集群中有 3 个 kube 节点.我有 3 个 Kafka 和 3 个 zookeeper 应用程序在运行，我有服务 zoo1、zoo2、zoo3 和 kafka-1、kafka-2 和 kafka-3 正在运行.我可以从 kubernetes 集群内部发布/使用，但我无法从 kubernetes 集群外部发布/使用，即从不属于 kubernetes 集群的外部机器发布/使用.
3. 我能够从外部机器访问 kube 节点 - 基本上我可以使用名称/IP ping 它们.
4. 我没有使用任何外部负载均衡器，但我有一个可以解析我的外部机器和 kube 节点的 DNS.
5. 在这种情况下，使用 NodePort 或 ExternalIP 公开 Kafka 服务不起作用.
6. 在 Kafka RC YML 中设置 KAFKA_ADVERTISED_HOST_NAME 或 KAFKA_ADVERTISED_LISTENERS，最终在 中设置 ADVERTISED_HOST_NAME/ADVERTISED_LISTENERS 属性server.properties 也无助于从 kubernetes 集群外部访问 kafka.

1. I have Kafka deployed and running in Kubernetes cluster. I am using this image from docker hub - https://hub.docker.com/r/cloudtrackinc/kubernetes-kafka/
2. I have 3 kube-nodes in my kubernetes cluster. I have 3 Kafka and 3 zookeeper applications running and I have services zoo1,zoo2,zoo3 and kafka-1, kafka-2 and kafka-3 running corresponding to them. I am able to publish/consume from inside kubernetes cluster but I am not able to publish/consume from outside of kubernetes cluster i.e., from external machine not part of kubernetes cluster.
3. I am able to reach the kube-nodes from external machine - basically I can ping them using name/ip.
4. I am not using any external load balancer but I have a DNS that can resolve both my external machine and kube-nodes.
5. Using NodePort or ExternalIP to expose the Kafka service does not work in this case.
6. Setting KAFKA_ADVERTISED_HOST_NAME or KAFKA_ADVERTISED_LISTENERS in Kafka RC YML that ultimately set ADVERTISED_HOST_NAME/ADVERTISED_LISTENERS properties in server.properties either does not help accessing kafka from outside of kubernetes cluster.

请建议我如何从 kubernetes 集群外部发布/使用.非常感谢！

Please suggest how can I publish/consume from outside of kubernetes cluster. Thanks much!

推荐答案

我在从 AWS 上的 k8s 集群外部访问 kafka 时遇到了同样的问题.我设法通过使用 0.10.2 版本支持多个接口的 kafka 侦听器功能来解决此问题.

I had the same problem with accessing kafka from outside of k8s cluster on AWS. I manage to solve this issue by using kafka listeners feature which from version 0.10.2 supports multiple interfaces.

这是我配置 kafka 容器的方式.

here is how I configured kafka container.

    ports:
    - containerPort: 9092
    - containerPort: 9093
    env:
    - name: KAFKA_ZOOKEEPER_CONNECT
      value: "zookeeper:2181"
    - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
      value: "INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT"
    - name: KAFKA_ADVERTISED_LISTENERS
      value: "INTERNAL_PLAINTEXT://kafka-internal-service:9092,EXTERNAL_PLAINTEXT://123.us-east-2.elb.amazonaws.com:9093"
    - name: KAFKA_LISTENERS
      value: "INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:9093"
    - name: KAFKA_INTER_BROKER_LISTENER_NAME
      value: "INTERNAL_PLAINTEXT"

除此之外，我配置了两个服务.一种用于内部(无头)&一种用于外部(LoadBalancer)通信.

Apart from that I configured two Services. One for internal(Headless) & one for external(LoadBalancer) communication.

希望这能节省人们的时间.

Hopefully this will save people's time.

这篇关于Kubernetes 集群中的 Kafka - 如何从 Kubernetes 集群外部发布/消费消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！