如何从公共计算机生成消息到专用网络上的 Kafka 安装? [英] How to produce messages from public computers to a Kafka installation on a private network?

问题描述

运行我的 Kafka 服务器的系统有两个 NIC，一个具有公共 IP (135.220.23.45)，另一个具有私有 IP (192.168.1.14).私有网卡连接到一个由 7 台机器组成的子网(地址均为 192.168.1.xxx).Kafka 已安装为使用 HDP 的服务，并已配置为 zookeeper.connect=192.168.1.14:2181 和 listeners=PLAINTEXT://192.168.1.14:6667.我使用以下命令在托管 kafka 服务器的系统上启动了一个使用者:[bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.14:6667 --topic test --from-beginning].

The system on which my Kafka server is running has two NICs, one with a public IP (135.220.23.45) and the other with a private one (192.168.1.14). The private NIC is connected to a subnet composed of 7 machines in total (all with addresses 192.168.1.xxx). Kafka has been installed as a service using HDP and has been configured with zookeeper.connect=192.168.1.14:2181 and listeners=PLAINTEXT://192.168.1.14:6667. I have started a consumer on the system that hosts the kafka server using: [bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.14:6667 --topic test --from-beginning].

当我在私有子网上的任何机器上启动生产者(使用 [bin/kafka-console-producer.sh --broker-list 192.168.1.14:6667 --topic test])时，消息被正常接收消费者.

When I start producers (using [bin/kafka-console-producer.sh --broker-list 192.168.1.14:6667 --topic test]) on any of the machines on the private subnet the messages are received normally by the consumer.

我想在公共系统上启动生产者并接收运行在 kafka 服务器上的消费者的消息.我相信这可以通过 IP 伪装和将所有外部请求转发到 135.220.23.45:15501(我选择 15501 来接收 kafka 消息)到 192.168.1.14:6667 来实现.为此，我在 firewalld 上设置了此端口转发规则:[port=15501:proto=tcp:toport=6670:toaddr=192.168.1.14].

I would like to start producers on public systems and receive the messages by the consumer running on the kafka server. I believed that this could be achieved by IP masquerading and by forwarding all external requests to 135.220.23.45:15501 (I have chosen 15501 to receive kafka messages) to 192.168.1.14:6667. To that extend I setup this port forwarding rule on firewalld: [port=15501:proto=tcp:toport=6670:toaddr=192.168.1.14].

但是，这似乎不起作用，因为当我使用 [bin/kafka-console-producer.sh --broker-list 135.220.23.45:15501 --topic] 在外部系统上启动生产者时，消息不能被消费者接收.

However, this doesn’t seem to work since when I start a producer on an external system with [bin/kafka-console-producer.sh --broker-list 135.220.23.45:15501 --topic] the messages cannot be received by the consumer.

我尝试了不同的 kafka 配置设置为 listeners 和adverted.listeners 但它们都不起作用.任何帮助将不胜感激.

I have tried different kafka config settings for listeners and advertised.listeners but none of them worked. Any help will be greatly appreciated.

推荐答案

您需要为内部和外部流量定义不同的端点才能使其正常工作.按照当前的配置，当您连接到 135.220.23.45:15501 时，Kafka 会回复请在 192.168.1.14:6667 上与我交谈，该地址无法从外部访问，并且从那里开始的所有内容都失败了.

You need to define different endpoints for your internal and external traffic in order for this to work. As it is currently configured, when you connect to 135.220.23.45:15501 Kafka would reply with "please talk to me on 192.168.1.14:6667 which is not reachable from the outside and everything from there on out fails.

使用 KIP-103 Kafka 被扩展以通过让您定义多个端点来满足这些场景.完全公开，我还没有尝试过这个，但按照以下思路至少应该让你走上正确的道路.

With KIP-103 Kafka was extended to cater to these scenarios by letting you define multiple endpoints. Full disclosure, I have not yet tried this out, but something along the following lines should at least get you started down the right road.

advertised.listeners=EXTERNAL://135.220.23.45:15501,INTERNAL://192.168.1.14:6667
inter.broker.listener.name=INTERNAL
listener.security.protocol.map=EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT

更新:

出于兴趣，我在由三台 ec2 机器组成的集群上对此进行了测试.我使用了以下配置:

I've tested this on a cluster of three ec2 machines out of interest. I've used the following configuration:

# internal ip: 172.31.61.130
# external ip: 184.72.211.109

listeners=INTERNAL://:9092,EXTERNAL_PLAINTEXT://:9094
advertised.listeners=INTERNAL://172.31.61.130:9092,EXTERNAL_PLAINTEXT://184.72.211.109:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT
inter.broker.listener.name=INTERNAL

这让我可以从内部机器和家里的笔记本电脑发送消息:

And that allowed me to send messages from both an internal machine as well as my laptop at home:

# Create topic 
kafka-topics --create --topic testtopic --partitions 9 --replication-factor 3 --zookeeper 127.0.0.1:2181

# Produce messages from internal machine
[ec2-user@ip-172-31-61-130 ~]$ kafka-console-producer --broker-list 127.0.0.1:9092 --topic testtopic                                                                                                               
>internal1
>internal2
>internal3

# Produce messages from external machine
➜  bin ./kafka-console-producer --topic testtopic --broker-list 184.72.211.109:9094
external1
external2
external3

# Check topic
[ec2-user@ip-172-31-61-130 ~]$ kafka-console-consumer --bootstrap-server 172.31.52.144:9092 --topic testtopic --from-beginning
external3
internal2
external1
external2
internal3
internal1

这篇关于如何从公共计算机生成消息到专用网络上的 Kafka 安装?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！