Kubernetes Cluster中的Kafka-如何从Kubernetes Cluster外部发布/使用消息 [英] Kafka in Kubernetes Cluster- How to publish/consume messages from outside of Kubernetes Cluster

问题描述

1. 我已经在Kubernetes集群中部署并运行了Kafka.我正在使用来自docker hub的图像- https://hub.docker.com/r/cloudtrackinc/kubernetes-kafka/
2. 我的kubernetes集群中有3个kube节点.我有3个Kafka和3个zookeeper应用程序正在运行，并且有对应于它们运行的​​服务zoo1，zoo2，zoo3和kafka-1，kafka-2和kafka-3.我可以从kubernetes集群内部发布/使用，但不能从kubernetes集群外部发布/使用，即从不属于kubernetes集群的外部机器发布/使用.
3. 我能够从外部计算机访问kube节点-基本上我可以使用name/ip对其执行ping操作.
4. 我没有使用任何外部负载平衡器，但是我有一个可以同时解析我的外部计算机和kube节点的DNS.
5. 在这种情况下，无法使用NodePort或ExternalIP公开Kafka服务.
最终在server.properties中设置ADVERTISED_HOST_NAME/ADVERTISED_LISTENERS属性的Kafka RC YML中的
6. 设置KAFKA_ADVERTISED_HOST_NAME或KAFKA_ADVERTISED_LISTENERS都不能帮助从kubernetes集群外部访问kafka.

1. I have Kafka deployed and running in Kubernetes cluster. I am using this image from docker hub - https://hub.docker.com/r/cloudtrackinc/kubernetes-kafka/
2. I have 3 kube-nodes in my kubernetes cluster. I have 3 Kafka and 3 zookeeper applications running and I have services zoo1,zoo2,zoo3 and kafka-1, kafka-2 and kafka-3 running corresponding to them. I am able to publish/consume from inside kubernetes cluster but I am not able to publish/consume from outside of kubernetes cluster i.e., from external machine not part of kubernetes cluster.
3. I am able to reach the kube-nodes from external machine - basically I can ping them using name/ip.
4. I am not using any external load balancer but I have a DNS that can resolve both my external machine and kube-nodes.
5. Using NodePort or ExternalIP to expose the Kafka service does not work in this case.
6. Setting KAFKA_ADVERTISED_HOST_NAME or KAFKA_ADVERTISED_LISTENERS in Kafka RC YML that ultimately set ADVERTISED_HOST_NAME/ADVERTISED_LISTENERS properties in server.properties either does not help accessing kafka from outside of kubernetes cluster.

请建议我如何从kubernetes集群外部发布/使用.非常感谢！

Please suggest how can I publish/consume from outside of kubernetes cluster. Thanks much!

推荐答案

我在从AWS上的k8s集群外部访问kafka时遇到了同样的问题.我设法通过使用Kafka侦听器功能解决此问题，该功能从0.10.2版本开始支持多个接口.

I had the same problem with accessing kafka from outside of k8s cluster on AWS. I manage to solve this issue by using kafka listeners feature which from version 0.10.2 supports multiple interfaces.

这是我配置kafka容器的方式.

here is how I configured kafka container.

    ports:
    - containerPort: 9092
    - containerPort: 9093
    env:
    - name: KAFKA_ZOOKEEPER_CONNECT
      value: "zookeeper:2181"
    - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
      value: "INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT"
    - name: KAFKA_ADVERTISED_LISTENERS
      value: "INTERNAL_PLAINTEXT://kafka-internal-service:9092,EXTERNAL_PLAINTEXT://123.us-east-2.elb.amazonaws.com:9093"
    - name: KAFKA_LISTENERS
      value: "INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:9093"
    - name: KAFKA_INTER_BROKER_LISTENER_NAME
      value: "INTERNAL_PLAINTEXT"

除此之外，我还配置了两个服务.一种用于内部(无头)和一个用于外部(LoadBalancer)通信.

Apart from that I configured two Services. One for internal(Headless) & one for external(LoadBalancer) communication.

希望这可以节省人们的时间.

Hopefully this will save people's time.

这篇关于Kubernetes Cluster中的Kafka-如何从Kubernetes Cluster外部发布/使用消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！