Kafka 快速入门，adverted.host.name 给出 kafka.common.LeaderNotAvailableException [英] Kafka QuickStart, advertised.host.name gives kafka.common.LeaderNotAvailableException

问题描述

我能够在一台 Linux 机器上本地运行一个简单的单节点 Kafka (kafka_2.11-0.8.2.1)，但是当我尝试远程运行生产者时，我遇到了一些令人困惑的错误.

我正在遵循 http://kafka.apache.org/documentation.html#quickstart.我停止了kafka进程并删除了所有zookeeper &/tmp 中的业力文件.我在本地 10.0.0.0/24 网络上使用外部 IP 地址进行 NAT，所以我修改了 server.properties 以告诉 zookeeper 如何广播我的外部地址，按照 https://medium.com/@thedude_rog/running-kafka-in-a-hybrid-cloud-environment-17a8f3cfc284:

advertised.host.name=MY.EXTERNAL.IP

然后我运行这个:

$ bin/zookeeper-server-start.sh config/zookeeper.properties-->...$ export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M" # 小测试服务器！$ bin/kafka-server-start.sh config/server.properties-->...

我在远程机器上为我的生产者打开了防火墙，并创建了一个新主题并验证了它:

$ bin/kafka-topics.sh --create --zookeeper MY.EXTERNAL.IP:2181 --replication-factor 1 --partitions 1 --topic test123-->创建主题test123".$ bin/kafka-topics.sh --list --zookeeper MY.EXTERNAL.IP:2181-->测试123

但是，我远程运行的生产者给了我错误:

$ bin/kafka-console-producer.sh --broker-list MY.EXTERNAL.IP:9092 --topic test123-->[2015-06-16 14:41:19,757] WARN 属性主题无效 (kafka.utils.VerifiableProperties)我的测试消息-->[2015-06-16 14:42:43,347] WARN 获取元数据时出错 [{TopicMetadata for topic test123 ->

由于主题 [test123] 的 kafka.common.LeaderNotAvailableException}，主题 test123 没有分区元数据:class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)-->(重复几次)

(我禁用了整个防火墙以确保这不是问题.)

karma-startup 中的 stdout 错误重复:[2015-06-16 20:42:42,768] INFO 正在关闭与/MY.EXTERNAL.IP 的套接字连接.(kafka.network.Processor)

controller.log 多次给我这个:

java.nio.channels.ClosedChannelException在 kafka.network.BlockingChannel.send(BlockingChannel.scala:100)在 kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)在 kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)在 kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)[2015-06-16 20:44:08,128] INFO [Controller-0-to-broker-0-send-thread], Controller 0 连接到 id:0,host:MY.EXTERNAL.IP,port:9092 用于发送状态更改请求(kafka.controller.RequestSendThread)[2015-06-16 20:44:08,428] WARN [Controller-0-to-broker-0-send-thread], Controller 0 epoch 1 无法发送请求 Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:7;ClientId:id_0-host_null-port_9092;Leaders:id:0,host:MY.EXTERNAL.IP,port:9092;PartitionState:(test123,0) ->(LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0) 到代理 id:0,host:MY.EXTERNAL.IP,port:9092.重新连接到代理.(kafka.controller.RequestSendThread)

运行这个好像表明0处有leader:

$ ./bin/kafka-topics.sh --zookeeper MY.EXTERNAL.IP:2181 --describe --topic test123-->主题:test123 PartitionCount:1 ReplicationFactor:1 配置:主题:test123 分区:0 领导者:0 副本:0 Isr:0

我重新运行了这个测试，我的 server.log 表明在 0 处有一个领导者:

<预><代码>...[2015-06-16 21:58:04,498] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)[2015-06-16 21:58:04,642] INFO 在路径/brokers/ids/0 注册代理 0，地址为 MY.EXTERNAL.IP:9092.(kafka.utils.ZkUtils$)[2015-06-16 21:58:04,670] INFO [Kafka Server 0]，启动(kafka.server.KafkaServer)[2015-06-16 21:58:04,736] INFO 新领导者为 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

当我从生产者发送消息时，我在日志中看到此错误:

[2015-06-16 22:18:24,584] ERROR [KafkaApi-0] 处理请求时出错 Name: TopicMetadataRequest;版本:0；相关 ID:7；ClientId:控制台生产者；主题:test123 (kafka.server.KafkaApis)kafka.admin.AdminOperationException:复制因子:1 比可用代理大:0在 kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)

我认为这意味着由于某种原因无法找到经纪人?我很困惑这是什么意思...

解决方案

总之，解决方案是通过 NAT 添加路由，以便机器可以访问自己的外部 IP 地址.

Zookeeper 使用它在 adsd.host.name 中找到的地址来告诉客户端在哪里可以找到代理以及与代理本身进行通信.报告的错误并没有说明这一点，而且令人困惑，因为客户端打开 TCP 连接没有问题.

I am able to get a simple one-node Kafka (kafka_2.11-0.8.2.1) working locally on one linux machine, but when I try to run a producer remotely I'm getting some confusing errors.

I'm following the quickstart guide at http://kafka.apache.org/documentation.html#quickstart. I stopped the kafka processes and deleted all the zookeeper & karma files in /tmp. I am on a local 10.0.0.0/24 network NAT-ed with an external IP address, so I modified server.properties to tell zookeeper how to broadcast my external address, as per https://medium.com/@thedude_rog/running-kafka-in-a-hybrid-cloud-environment-17a8f3cfc284:

advertised.host.name=MY.EXTERNAL.IP

Then I'm running this:

$ bin/zookeeper-server-start.sh config/zookeeper.properties
--> ...
$ export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M" # small test server!
$ bin/kafka-server-start.sh config/server.properties
--> ...

I opened up the firewall for my producer on the remote machine, and created a new topic and verified it:

$ bin/kafka-topics.sh --create --zookeeper MY.EXTERNAL.IP:2181 --replication-factor 1 --partitions 1 --topic test123
--> Created topic "test123".
$ bin/kafka-topics.sh --list --zookeeper MY.EXTERNAL.IP:2181
--> test123

However, the producer I'm running remotely gives me errors:

$ bin/kafka-console-producer.sh --broker-list MY.EXTERNAL.IP:9092 --topic test123
--> [2015-06-16 14:41:19,757] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
My Test Message
--> [2015-06-16 14:42:43,347] WARN Error while fetching metadata [{TopicMetadata for topic test123 -> 

No partition metadata for topic test123 due to kafka.common.LeaderNotAvailableException}] for topic [test123]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) --> (repeated several times)

(I disabled the whole firewall to make sure that wasn't the problem.)

The stdout errors in the karma-startup are repeated: [2015-06-16 20:42:42,768] INFO Closing socket connection to /MY.EXTERNAL.IP. (kafka.network.Processor)

And the controller.log gives me this, several times:

java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
    at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
[2015-06-16 20:44:08,128] INFO [Controller-0-to-broker-0-send-thread], Controller 0 connected to id:0,host:MY.EXTERNAL.IP,port:9092 for sending state change requests (kafka.controller.RequestSendThread)
[2015-06-16 20:44:08,428] WARN [Controller-0-to-broker-0-send-thread], Controller 0 epoch 1 fails to send request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:7;ClientId:id_0-host_null-port_9092;Leaders:id:0,host:MY.EXTERNAL.IP,port:9092;PartitionState:(test123,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0) to broker id:0,host:MY.EXTERNAL.IP,port:9092. Reconnecting to broker. (kafka.controller.RequestSendThread)

Running this seems to indicate that there is a leader at 0:

$ ./bin/kafka-topics.sh --zookeeper MY.EXTERNAL.IP:2181 --describe --topic test123
--> Topic:test123   PartitionCount:1    ReplicationFactor:1 Configs:
Topic: test123  Partition: 0    Leader: 0   Replicas: 0 Isr: 0

I reran this test and my server.log indicates that there is a leader at 0:

...
[2015-06-16 21:58:04,498] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2015-06-16 21:58:04,642] INFO Registered broker 0 at path /brokers/ids/0 with address MY.EXTERNAL.IP:9092. (kafka.utils.ZkUtils$)
[2015-06-16 21:58:04,670] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2015-06-16 21:58:04,736] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

I see this error in the logs when I send a message from the producer:

[2015-06-16 22:18:24,584] ERROR [KafkaApi-0] error when handling request Name: TopicMetadataRequest; Version: 0; CorrelationId: 7; ClientId: console-producer; Topics: test123 (kafka.server.KafkaApis)
kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0
    at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)

I assume this means that the broker can't be found for some reason? I'm confused what this means...

解决方案

So to summarize, the solution to this was to add a route via NAT so that the machine can access its own external IP address.

Zookeeper uses the address it finds in advertised.host.name both to tell clients where to find the broker as well as to communicate with the broker itself. The error that gets reported doesn't make this very clear, and it's confusing because a client has no problem opening a TCP connection.

这篇关于Kafka 快速入门，adverted.host.name 给出 kafka.common.LeaderNotAvailableException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！