远程访问在kubernetes中运行的Kafka [英] Remotely accessing Kafka running inside kubernetes
问题描述
我有一个在单节点kubernetes环境上的pod内运行的单节点Kafka代理.我将这张图片用于kafka: https://hub.docker.com/r/wurstmeister/卡夫卡
I have a single node Kafka broker running inside a pod on a single node kubernetes environment. I am using this image for kafka: https://hub.docker.com/r/wurstmeister/kafka
kafka版本 = 1.1.0
Kubernetes集群在服务器上的VM内部运行. VM在活动接口ens32上具有以下IP-192.168.3.102
Kubernetes cluster is running inside a VM on a server. The VM has the following IP on the active interface ens32 - 192.168.3.102
Kafka.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: casb-deployment
name: kafkaservice
spec:
replicas: 1
template:
metadata:
labels:
app: kafkaservice
spec:
hostname: kafkaservice
containers:
- name: kafkaservice
imagePullPolicy: IfNotPresent
image: wurstmeister/kafka:1.1.0
env:
- name: KAFKA_BROKER_ID
value: "1"
# - name: KAFKA_ADVERTISED_HOST_NAME
# value: "kafkaservice"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT"
- name: KAFKA_LISTENERS
value: "INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:9093"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INTERNAL_PLAINTEXT://kafkaservice:9092,EXTERNAL_PLAINTEXT://192.168.3.102:9093"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INTERNAL_PLAINTEXT"
- name: KAFKA_CREATE_TOPICS
value: "topic-1:100:1,topic-2:1:1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper:2181"
ports:
- name: port9092
containerPort: 9092
- name: port9093
containerPort: 9093
volumeMounts:
- mountPath: /kafka/kafka-logs-kafkaservice
name: kafka-volume
volumes:
- name: kafka-volume
hostPath:
path: /home/volume/kafka-logs
---
apiVersion: v1
kind: Service
metadata:
namespace: casb-deployment
name: kafkaservice
labels:
app: kafkaservice
spec:
selector:
app: kafkaservice
ports:
- name: port9092
port: 9092
targetPort: 9092
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
namespace: casb-deployment
name: kafkaservice-external
labels:
app: kafkaservice-external
spec:
selector:
app: kafkaservice
ports:
- name: port9093
port: 9093
protocol: TCP
nodePort: 30035
type: NodePort
我能够从本地计算机ping 192.168.3.102 ping VM，即kubernetes节点，并且正在使用nodePort公开服务.
I am able to ping the VM i.e. the kubernetes node from my local machine ping 192.168.3.102 and I am using nodePort to expose the service.
我也可以telnet telnet 192.168.3.102 30035，它给出了:
I can also telnet telnet 192.168.3.102 30035 and it gives:
Trying 192.168.3.102...
Connected to 192.168.3.102.
Escape character is '^]'.
我尝试从本地计算机上运行kafka控制台使用者和生产者:
I tried running kafka console consumer and producer from my local machine:
消费者: bin/kafka-console-consumer.sh --bootstrap-server 192.168.3.102:30035 --topic topic-1 --from-beginning
Consumer: bin/kafka-console-consumer.sh --bootstrap-server 192.168.3.102:30035 --topic topic-1 --from-beginning
输出:
[2019-09-25 12:30:40,716] WARN [Consumer clientId=consumer-1, groupId=console-consumer-20551] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
制作人:
bin/kafka-console-producer.sh --broker-list 192.168.3.102:30035 --topic topic-1
输出:
[2019-09-25 12:32:07,958] WARN [Producer clientId=console-producer] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Netstat:
netstat -tulpn | grep 30035
tcp6 0 0 :::30035 :::* LISTEN 113545/kube-proxy
我尝试运行基于python的使用者，即 kafka-python == 1.4.2 它给了我以下日志:
I tried running a python based consumer i.e kafka-python==1.4.2 it gave me the following logs:
[2019-09-25T12:15:39+0500] INFO kafka.client Bootstrapping cluster metadata from [('192.168.3.102', 30035, <AddressFamily.AF_INET: 2>)]
[2019-09-25T12:15:39+0500] INFO kafka.conn <BrokerConnection node_id=bootstrap host=192.168.3.102:30035 <connecting> [IPv4 ('192.168.3.102', 30035)]>: connecting to 192.168.3.102:30035 [('192.168.3.102', 30035) IPv4]
[2019-09-25T12:15:39+0500] INFO kafka.conn <BrokerConnection node_id=bootstrap host=192.168.3.102:30035 <connecting> [IPv4 ('192.168.3.102', 30035)]>: Connection complete.
[2019-09-25T12:15:39+0500] INFO kafka.client Bootstrap succeeded: found 1 brokers and 26 topics.
[2019-09-25T12:15:39+0500] INFO kafka.conn <BrokerConnection node_id=bootstrap host=192.168.3.102:30035 <connected> [IPv4 ('192.168.3.102', 30035)]>: Closing connection.
[2019-09-25T12:15:39+0500] INFO kafka.conn <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]>: connecting to 192.168.3.102:9093 [('192.168.3.102', 9093) IPv4]
[2019-09-25T12:15:39+0500] INFO kafka.conn Probing node 1 broker version
[2019-09-25T12:15:39+0500] ERROR kafka.conn Connect attempt to <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]> returned error 111. Disconnecting.
[2019-09-25T12:15:39+0500] INFO kafka.conn <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]>: Closing connection. ConnectionError: 111 ECONNREFUSED
[2019-09-25T12:15:40+0500] INFO kafka.conn <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]>: connecting to 192.168.3.102:9093 [('192.168.3.102', 9093) IPv4]
[2019-09-25T12:15:40+0500] ERROR kafka.conn Connect attempt to <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]> returned error 111. Disconnecting.
[2019-09-25T12:15:40+0500] INFO kafka.conn <BrokerConnection node_id=1 host=192.168.3.102:9093 <connecting> [IPv4 ('192.168.3.102', 9093)]>: Closing connection. ConnectionError: 111 ECONNREFUSED
[2019-09-25T12:15:40+0500] INFO Activity URL collector Exception in activity url collector: NoBrokersAvailable
从日志看来，就是建立了连接.
From the logs it seems like the connection was made i.e.
<connecting> [IPv4 ('192.168.3.102', 30035)]>: Connection complete.
Bootstrap succeeded: found 1 brokers and 26 topics.
但是后来它断开了连接.
But then it got disconnected.
请帮助我弄清楚我想念的是什么以及如何解决这个问题.谢谢.
Please help me out in figuring out what am I missing and how can I resolve this. Thanks.
推荐答案
您正在引导至端口30035，并获得了初始连接，然后返回了广告的9093端口以用于后续连接，而不是30035.
You're bootstrapping to port 30035, and getting the initial connection, then the advertised port of 9093 is being returned for subsequent connections, not 30035.
您需要NodePort和播发的NodePort是相同的，或者至少两者都可以在外部路由.然后，如果您的代码在主机上运行，那么您还需要在VM上进行端口转发
You need the NodePort and advertised one to be the same, or at least both be externally routable. Then you additionally need to have port forwarding on your VM if your code is running on your host machine
注意:存在用于在Kubernetes中设置Kafka的合流或Strimzi舵图
Note: Confluent or Strimzi Helm Charts exist for setting up Kafka in Kubernetes
这篇关于远程访问在kubernetes中运行的Kafka的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
