Kafka 服务器配置 - 监听器 vs.advertised.listeners [英] Kafka server configuration - listeners vs. advertised.listeners

问题描述

要让 Kafka 运行，您需要在 config/server.properties 文件中设置一些属性.有两个设置我不明白.

To get Kafka running, you need to set some properties in config/server.properties file. There are two settings I don't understand.

有人能解释一下 listeners 和 Advertising.listeners 属性之间的区别吗?

Can somebody explain the difference between listeners and advertised.listeners property?

文档说:

listeners:socket 服务器监听的地址.

listeners: The address the socket server listens on.

和

advertised.listeners:代理将向生产者和消费者宣传的主机名和端口.

advertised.listeners: Hostname and port the broker will advertise to producers and consumers.

我什么时候必须使用哪个设置?

When do I have to use which setting?

推荐答案

由于我还不能发表评论，我会将其作为答案"发布，添加到 M.Situations 答案中.

Since I cannot comment yet I will post this as an "answer", adding on to M.Situations answer.

在他链接的同一个文档中，有一个关于 KAFKA 客户端使用哪个侦听器的简介(https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+交通):

Within the same document he links there is this blurb about which listener is used by a KAFKA client (https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic):

如前所述，客户端永远不会看到侦听器名称，并且会像以前一样发出元数据请求.不同之处在于它们返回的端点列表仅限于它们发出请求的端点的侦听器名称.

As stated previously, clients never see listener names and will make metadata requests exactly as before. The difference is that the list of endpoints they get back is restricted to the listener name of the endpoint where they made the request.

这很重要，因为这取决于您在 bootstrap.servers 配置中使用的 URL*，如果它映射到 adsed.listeners 中，客户端将返回的 URL*(不知道如果侦听器的行为是什么不存在).

This is important as depending on what URL you use in your bootstrap.servers config that will be the URL* that the client will get back if it is mapped in advertised.listeners (do not know what the behavior is if the listener does not exist).

还要注意这一点:

基于 ZooKeeper 的消费者除外.这些消费者直接从 ZooKeeper 检索代理注册信息，并将选择第一个以 PLAINTEXT 作为安全协议(他们支持的唯一安全协议)的侦听器.

The exception is ZooKeeper-based consumers. These consumers retrieve the broker registration information directly from ZooKeeper and will choose the first listener with PLAINTEXT as the security protocol (the only security protocol they support).

作为代理配置示例(适用于集群中的所有代理):

As an example broker config (for all brokers in cluster):

advertised.listeners=EXTERNAL://XXXXX.compute-1.amazonaws.com:9990,INTERNAL://ip-XXXXX.ec2.internal:9993

advertised.listeners=EXTERNAL://XXXXX.compute-1.amazonaws.com:9990,INTERNAL://ip-XXXXX.ec2.internal:9993

inter.broker.listener.name=内部

inter.broker.listener.name=INTERNAL

listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:PLAINTEXT

listener.security.protocol.map=EXTERNAL:SSL,INTERNAL:PLAINTEXT

如果客户端使用 XXXXX.compute-1.amazonaws.com:9990 进行连接，则元数据提取将转到该代理.但是，与组协调员或领导者一起使用的返回 URL 可能是 123.compute-1.amazonaws.com:9990*(不同的机器！).这意味着匹配是在 KIP-103 宣传的侦听器名称上完成的，而不管实际 URL(节点)如何.

If the client uses XXXXX.compute-1.amazonaws.com:9990 to connect, the metadata fetch will go to that broker. However, the returning URL to use with the Group Coordinator or Leader could be 123.compute-1.amazonaws.com:9990* (a different machine!). This means that the match is done on the listener name as advertised by KIP-103 irrespective of the actual URL (node).

由于 EXTERNAL 的协议映射是 SSL，这将强制您使用 SSL 密钥库进行连接.

Since the protocol map for EXTERNAL is SSL this would force you to use an SSL keystore to connect.

另一方面，如果您在 AWS 内，那么您可以发出 ip-XXXXX.ec2.internal:9993 并且相应的连接将根据协议映射为明文.

If on the other hand you are within AWS lets say, you can then issue ip-XXXXX.ec2.internal:9993 and the corresponding connection would be plaintext as per the protocol map.

这在 IaaS 中尤其需要，在我的案例中，代理和消费者位于 AWS 上，而我的生产者位于客户端站点，因此需要不同的安全协议和侦听器.

This is especially needed in IaaS where in my case brokers and consumers live on AWS, whereas my producer lives on a client site, thus needing different security protocols and listeners.

由于您为不同的客户端(代理、生产者、消费者)拥有不同的端口，因此添加入站规则也变得更加容易.

Also adding Inbound Rules is much easier now that you have different ports for different clients (brokers, producers, consumers).

如果以上内容仍然不清楚，这篇文章是一个很好的深度指南:https://rmoff.net/2018/08/02/kafka-listeners-explained/

This article is a great in depth guide if the above is still not clear: https://rmoff.net/2018/08/02/kafka-listeners-explained/

这篇关于Kafka 服务器配置 - 监听器 vs.advertised.listeners的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！