如何从公用计算机到专用网络上的Kafka安装产生消息? [英] How to produce messages from public computers to a Kafka installation on a private network?

问题描述

运行我的Kafka服务器的系统有两个NIC，一个具有公共IP(135.220.23.45)，另一个具有私有IP(192.168.1.14).专用NIC连接到总共由7台计算机组成的子网(所有地址均为192.168.1.xxx). Kafka已使用HDP作为服务安装，并已通过zookeeper.connect = 192.168.1.14:2181和listeners = PLAINTEXT://192.168.1.14:6667配置.我已经在使用以下命令托管kafka服务器的系统上启动了一个使用者:[bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.14:6667 --topic test --from-beginning].

The system on which my Kafka server is running has two NICs, one with a public IP (135.220.23.45) and the other with a private one (192.168.1.14). The private NIC is connected to a subnet composed of 7 machines in total (all with addresses 192.168.1.xxx). Kafka has been installed as a service using HDP and has been configured with zookeeper.connect=192.168.1.14:2181 and listeners=PLAINTEXT://192.168.1.14:6667. I have started a consumer on the system that hosts the kafka server using: [bin/kafka-console-consumer.sh --bootstrap-server 192.168.1.14:6667 --topic test --from-beginning].

当我在专用子网上的任何机器上启动生产者(使用[bin/kafka-console-producer.sh --broker-list 192.168.1.14:6667 --topic test])时，消息通常会被正常接收消费者.

When I start producers (using [bin/kafka-console-producer.sh --broker-list 192.168.1.14:6667 --topic test]) on any of the machines on the private subnet the messages are received normally by the consumer.

我想在公共系统上启动生产者，并接收运行在kafka服务器上的消费者的消息.我相信可以通过伪装IP并将所有外部请求转发到135.220.23.45:15501(我已选择15501来接收kafka消息)到192.168.1.14:6667来实现.为此，我在firewalld上设置了此端口转发规则:[port = 15501:proto = tcp:toport = 6670:toaddr = 192.168.1.14].

I would like to start producers on public systems and receive the messages by the consumer running on the kafka server. I believed that this could be achieved by IP masquerading and by forwarding all external requests to 135.220.23.45:15501 (I have chosen 15501 to receive kafka messages) to 192.168.1.14:6667. To that extend I setup this port forwarding rule on firewalld: [port=15501:proto=tcp:toport=6670:toaddr=192.168.1.14].

但是，这似乎不起作用，因为当我使用[bin/kafka-console-producer.sh --broker-list 135.220.23.45:15501 --topic]在外部系统上启动生产者时，消息无法受到消费者的欢迎.

However, this doesn’t seem to work since when I start a producer on an external system with [bin/kafka-console-producer.sh --broker-list 135.220.23.45:15501 --topic] the messages cannot be received by the consumer.

我已经为监听器和advertised.listener尝试了不同的kafka配置设置，但是没有一个起作用.任何帮助将不胜感激.

I have tried different kafka config settings for listeners and advertised.listeners but none of them worked. Any help will be greatly appreciated.

推荐答案

您需要为内部和外部流量定义不同的端点，以使其正常工作.按照目前的配置，当您连接到135.220.23.45:15501时，Kafka会回复请在192.168.1.14:6667上与我交谈，这是无法从外部访问的，此后一切都会失败.

You need to define different endpoints for your internal and external traffic in order for this to work. As it is currently configured, when you connect to 135.220.23.45:15501 Kafka would reply with "please talk to me on 192.168.1.14:6667 which is not reachable from the outside and everything from there on out fails.

使用 KIP-103 Kafka已扩展，可以通过定义多个端点来满足这些情况. 完全公开，我还没有尝试过，但是以下几方面的内容至少应该使您踏上正确的道路.

With KIP-103 Kafka was extended to cater to these scenarios by letting you define multiple endpoints. Full disclosure, I have not yet tried this out, but something along the following lines should at least get you started down the right road.

advertised.listeners=EXTERNAL://135.220.23.45:15501,INTERNAL://192.168.1.14:6667
inter.broker.listener.name=INTERNAL
listener.security.protocol.map=EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT

更新:

我已经在不感兴趣的三台ec2机器的群集上对此进行了测试.我使用了以下配置:

I've tested this on a cluster of three ec2 machines out of interest. I've used the following configuration:

# internal ip: 172.31.61.130
# external ip: 184.72.211.109

listeners=INTERNAL://:9092,EXTERNAL_PLAINTEXT://:9094
advertised.listeners=INTERNAL://172.31.61.130:9092,EXTERNAL_PLAINTEXT://184.72.211.109:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT
inter.broker.listener.name=INTERNAL

这使我既可以从内部计算机也可以从家里的笔记本电脑发送消息:

And that allowed me to send messages from both an internal machine as well as my laptop at home:

# Create topic 
kafka-topics --create --topic testtopic --partitions 9 --replication-factor 3 --zookeeper 127.0.0.1:2181

# Produce messages from internal machine
[ec2-user@ip-172-31-61-130 ~]$ kafka-console-producer --broker-list 127.0.0.1:9092 --topic testtopic                                                                                                               
>internal1
>internal2
>internal3

# Produce messages from external machine
➜  bin ./kafka-console-producer --topic testtopic --broker-list 184.72.211.109:9094
external1
external2
external3

# Check topic
[ec2-user@ip-172-31-61-130 ~]$ kafka-console-consumer --bootstrap-server 172.31.52.144:9092 --topic testtopic --from-beginning
external3
internal2
external1
external2
internal3
internal1

这篇关于如何从公用计算机到专用网络上的Kafka安装产生消息?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！