通过 SSH 隧道从 Kafka 集群消费 [英] Consume from a Kafka Cluster through SSH Tunnel

问题描述

我们正在尝试使用 Java 客户端从 Kafka 集群中消费.集群位于 Jump 主机后面，因此访问的唯一方法是通过 SSH 隧道.但是我们无法读取，因为一旦消费者获取元数据，它就会使用原始主机连接到代理.这种行为可以被覆盖吗?我们可以要求 Kafka Client 不使用元数据吗?

We are trying to consume from a Kafka Cluster using the Java Client. The Cluster is a behind a Jump host and hence the only way to access is through a SSH Tunnel. But we are not able read because once the consumer fetches metadata it uses the original hosts to connect to brokers. Can this behaviour be overridden? Can we ask Kafka Client to not use the metadata?

推荐答案

据我所知.

当我需要做类似的事情时，我使用的技巧是:

The trick I used when I needed to do something similar was:

1. 为每个 Kafka 代理设置一个虚拟接口
2. 为每个代理打开一个隧道，以便代理 n 绑定到虚拟接口 n
3. 配置您的 /etc/hosts 文件，以便将代理 n 的广告主机名解析为虚拟接口 n 的 IP.

1. setup a virtual interface for each Kafka broker
2. open a tunnel to each broker so that broker n is bound to virtual interface n
3. configure your /etc/hosts file so that the advertised hostname of broker n is resolved to the ip of the virtual interface n.

对.

Kafka 经纪人:

• broker1(宣传为 broker1.mykafkacluster)
• broker2(宣传为 broker2.mykafkacluster)

虚拟界面:

• veth1 (192.168.1.1)
• veth2 (192.168.1.2)

隧道:

• broker1:ssh -L 192.168.1.1:9092:broker1.mykafkacluster:9092 jumphost
• broker2:ssh -L 192.168.1.2:9092:broker1.mykafkacluster:9092 jumphost

/etc/hosts:

• 192.168.1.1 broker1.mykafkacluster
• 192.168.1.2 broker2.mykafkacluster

如果你像这样配置你的系统，你应该能够访问你的 Kafka 集群中的所有代理.

If you configure your system like this you should be able reach all the brokers in your Kafka cluster.

注意:如果您将 Kafka 代理配置为公布 IP 地址而不是主机名，该过程仍然可以工作，但您需要使用代理公布的相同 IP 地址配置虚拟接口.

Note: if you configured your Kafka brokers to advertise an ip address instead of a hostname the procedure can still work but you need to configure the virtual interfaces with the same ip address that the broker advertises.

这篇关于通过 SSH 隧道从 Kafka 集群消费的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！