Kafka 快速入门，advertised.host.name 给出 kafka.common.LeaderNotAvailableException [英] Kafka QuickStart, advertised.host.name gives kafka.common.LeaderNotAvailableException

问题描述

我能够在一台 Linux 机器上本地运行一个简单的单节点 Kafka (kafka_2.11-0.8.2.1)，但是当我尝试远程运行生产者时，我遇到了一些令人困惑的错误.

I am able to get a simple one-node Kafka (kafka_2.11-0.8.2.1) working locally on one linux machine, but when I try to run a producer remotely I'm getting some confusing errors.

我正在遵循 http://kafka.apache.org/documentation.html#quickstart.我停止了kafka进程并删除了所有zookeeper &/tmp 中的业力文件.我在本地 10.0.0.0/24 网络上使用外部 IP 地址进行 NAT，所以我修改了 server.properties 以告诉 zookeeper 如何广播我的外部地址，按照 https://medium.com/@thedude_rog/running-kafka-in-a-hybrid-cloud-environment-17a8f3cfc284:

I'm following the quickstart guide at http://kafka.apache.org/documentation.html#quickstart. I stopped the kafka processes and deleted all the zookeeper & karma files in /tmp. I am on a local 10.0.0.0/24 network NAT-ed with an external IP address, so I modified server.properties to tell zookeeper how to broadcast my external address, as per https://medium.com/@thedude_rog/running-kafka-in-a-hybrid-cloud-environment-17a8f3cfc284:

advertised.host.name=MY.EXTERNAL.IP

然后我运行这个:

$ bin/zookeeper-server-start.sh config/zookeeper.properties
--> ...
$ export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M" # small test server!
$ bin/kafka-server-start.sh config/server.properties
--> ...

我在远程机器上为我的生产者打开了防火墙，并创建了一个新主题并验证了它:

I opened up the firewall for my producer on the remote machine, and created a new topic and verified it:

$ bin/kafka-topics.sh --create --zookeeper MY.EXTERNAL.IP:2181 --replication-factor 1 --partitions 1 --topic test123
--> Created topic "test123".
$ bin/kafka-topics.sh --list --zookeeper MY.EXTERNAL.IP:2181
--> test123

但是，我远程运行的生产者给了我错误:

However, the producer I'm running remotely gives me errors:

$ bin/kafka-console-producer.sh --broker-list MY.EXTERNAL.IP:9092 --topic test123
--> [2015-06-16 14:41:19,757] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
My Test Message
--> [2015-06-16 14:42:43,347] WARN Error while fetching metadata [{TopicMetadata for topic test123 -> 

由于 kafka.common.LeaderNotAvailableException}，主题 [test123] 的主题 test123 没有分区元数据:类 kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)-->(重复几次)

No partition metadata for topic test123 due to kafka.common.LeaderNotAvailableException}] for topic [test123]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) --> (repeated several times)

(我禁用了整个防火墙以确保这不是问题.)

(I disabled the whole firewall to make sure that wasn't the problem.)

karma-startup 中的 stdout 错误重复:[2015-06-16 20:42:42,768] INFO 正在关闭与/MY.EXTERNAL.IP 的套接字连接.(kafka.network.Processor)

The stdout errors in the karma-startup are repeated: [2015-06-16 20:42:42,768] INFO Closing socket connection to /MY.EXTERNAL.IP. (kafka.network.Processor)

controller.log 多次给我这个:

And the controller.log gives me this, several times:

java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
    at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
    at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
[2015-06-16 20:44:08,128] INFO [Controller-0-to-broker-0-send-thread], Controller 0 connected to id:0,host:MY.EXTERNAL.IP,port:9092 for sending state change requests (kafka.controller.RequestSendThread)
[2015-06-16 20:44:08,428] WARN [Controller-0-to-broker-0-send-thread], Controller 0 epoch 1 fails to send request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:7;ClientId:id_0-host_null-port_9092;Leaders:id:0,host:MY.EXTERNAL.IP,port:9092;PartitionState:(test123,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0) to broker id:0,host:MY.EXTERNAL.IP,port:9092. Reconnecting to broker. (kafka.controller.RequestSendThread)

运行这个好像说明0处有leader:

Running this seems to indicate that there is a leader at 0:

$ ./bin/kafka-topics.sh --zookeeper MY.EXTERNAL.IP:2181 --describe --topic test123
--> Topic:test123   PartitionCount:1    ReplicationFactor:1 Configs:
Topic: test123  Partition: 0    Leader: 0   Replicas: 0 Isr: 0

我重新运行了这个测试，我的 server.log 表明在 0 处有一个领导者:

I reran this test and my server.log indicates that there is a leader at 0:

...
[2015-06-16 21:58:04,498] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2015-06-16 21:58:04,642] INFO Registered broker 0 at path /brokers/ids/0 with address MY.EXTERNAL.IP:9092. (kafka.utils.ZkUtils$)
[2015-06-16 21:58:04,670] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2015-06-16 21:58:04,736] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

当我从生产者发送消息时，我在日志中看到此错误:

I see this error in the logs when I send a message from the producer:

[2015-06-16 22:18:24,584] ERROR [KafkaApi-0] error when handling request Name: TopicMetadataRequest; Version: 0; CorrelationId: 7; ClientId: console-producer; Topics: test123 (kafka.server.KafkaApis)
kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0
    at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)

我认为这意味着由于某种原因无法找到经纪人?我很困惑这是什么意思...

I assume this means that the broker can't be found for some reason? I'm confused what this means...

推荐答案

对于 Kafka 的最新版本(在撰写本文时为 0.10.0)，您不想使用 advertised.host.name.事实上，即使 [documentation] 声明 advertised.host.name 已经被弃用.此外，Kafka 不仅会将它用作生产者/消费者的广告"主机名，还会用作其他代理(在多代理环境中)……这是一种如果您为代理使用不同的(可能是内部的)DNS，那会很痛苦……而且您真的不想从事将条目添加到各个 /etc/hosts 的业务经纪人(啊！)

For the recent versions of Kafka (0.10.0 as of this writing), you don't want to use advertised.host.name at all. In fact, even the [documentation] states that advertised.host.name is already deprecated. Moreover, Kafka will use this not only as the "advertised" host name for the producers/consumers, but for other brokers as well (in a multi-broker environment)...which is kind of a pain if you're using using a different (perhaps internal) DNS for the brokers...and you really don't want to get into the business of adding entries to the individual /etc/hosts of the brokers (ew!)

因此，基本上，您希望代理使用内部名称，但对生产者和消费者使用外部 FQDN 仅.为此，您将改为更新 advertised.listeners.

So, basically, you would want the brokers to use the internal name, but use the external FQDNs for the producers and consumers only. To do this, you will update advertised.listeners instead.

这篇关于Kafka 快速入门，advertised.host.name 给出 kafka.common.LeaderNotAvailableException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！