无法在 CentOS 7 上启动 kafka 代理 [英] unable to bring up kafka broker on CentOS 7

问题描述

Kafka 新手，我尝试在 Centos 服务器上安装 kafka，但出现以下错误，这是全新安装，我还没有配置任何东西来将数据推送到 kafka 生产者

New to Kafka ,I tried installing kafka on Centos server with the below error ,it's a fresh installation and I haven't configured anything to push data to the kafka producer

[2016-12-12 18:20:18,531] INFO zookeeper state changed (SyncConnected)     (org.I0Itec.zkclient.ZkClient)
[2016-12-12 18:20:18,601] INFO Loading logs. (kafka.log.LogManager)
[2016-12-12 18:20:18,609] INFO Logs loading complete. (kafka.log.LogManager)
[2016-12-12 18:20:18,676] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2016-12-12 18:20:18,678] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-12-12 18:20:18,734] INFO Awaiting socket connections on 172.16.23.13:9092. (kafka.network.Acceptor)
[2016-12-12 18:20:18,770] INFO [Socket Server on Broker 0], Started 1 acceptor threads (kafka.network.SocketServer)
[2016-12-12 18:20:18,796] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:20:18,798] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:20:18,876] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:20:18,887] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:20:18,888] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-12-12 18:20:18,980] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:20:18,981] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:20:18,995] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.GroupCoordinator)
[2016-12-12 18:20:18,996] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.GroupCoordinator)
[2016-12-12 18:20:19,001] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 10 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-12 18:20:19,016] INFO [ThrottledRequestReaper-Produce], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-12 18:20:19,017] INFO [ThrottledRequestReaper-Fetch], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-12 18:20:19,024] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2016-12-12 18:20:19,053] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:20:19,056] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:20:19,058] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(172.16.23.13,9092,PLAINTEXT) (kafka.utils.ZkUtils)
[2016-12-12 18:20:19,079] INFO Kafka version : 0.10.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2016-12-12 18:20:19,079] INFO Kafka commitId : b8642491e78c5a13 (org.apache.kafka.common.utils.AppInfoParser)
[2016-12-12 18:20:19,080] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-12-12 18:20:19,082] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2016-12-12 18:20:20,170] ERROR Processor got uncaught exception. (kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at    org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:323)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
    at kafka.network.Processor.poll(SocketServer.scala:472)
    at kafka.network.Processor.run(SocketServer.scala:412)
    at java.lang.Thread.run(Thread.java:745)
[2016-12-12 18:20:20,180] WARN Unexpected error from /172.16.23.13; closing connection (org.apache.kafka.common.network.Selector)
java.lang.NullPointerException
    at org.apache.kafka.common.network.NetworkReceive.complete(NetworkReceive.java:67)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:136)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:323)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
    at kafka.network.Processor.poll(SocketServer.scala:472)
    at kafka.network.Processor.run(SocketServer.scala:412)
    at java.lang.Thread.run(Thread.java:745)    

如果我增加内存，则会出现以下错误.

[2016-12-12 18:18:20,789] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2016-12-12 18:18:20,844] INFO Awaiting socket connections on 172.16.23.13:9092. (kafka.network.Acceptor)
[2016-12-12 18:18:20,848] INFO [Socket Server on Broker 0], Started 1 acceptor threads (kafka.network.SocketServer)
[2016-12-12 18:18:20,905] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:18:20,908] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:18:20,988] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:18:20,999] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:18:21,000] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2016-12-12 18:18:21,097] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:18:21,099] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2016-12-12 18:18:21,112] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.GroupCoordinator)
[2016-12-12 18:18:21,113] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.GroupCoordinator)
[2016-12-12 18:18:21,117] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 10 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-12-12 18:18:21,133] INFO [ThrottledRequestReaper-Produce], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-12 18:18:21,134] INFO [ThrottledRequestReaper-Fetch], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2016-12-12 18:18:21,141] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2016-12-12 18:18:21,168] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:18:21,172] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2016-12-12 18:18:21,174] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(172.16.23.13,9092,PLAINTEXT)  (kafka.utils.ZkUtils)
[2016-12-12 18:18:21,194] INFO Kafka version : 0.10.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2016-12-12 18:18:21,194] INFO Kafka commitId : b8642491e78c5a13 (org.apache.kafka.common.utils.AppInfoParser)
[2016-12-12 18:18:21,195] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-12-12 18:18:21,200] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2016-12-12 18:18:30,013] WARN Unexpected error from /172.16.23.13; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1347375956 larger than 104857600)
    at  org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:91)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:323)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
    at kafka.network.Processor.poll(SocketServer.scala:472)
    at kafka.network.Processor.run(SocketServer.scala:412)
    at java.lang.Thread.run(Thread.java:745)

推荐答案

其他一些进程正在向 kafka 发送数据.在我的例子中，influxdb 正在向 kafka 端口发送数据，因为 kapacitor 也监听 9092 端口.所以 influxdb 正在向 kafka 发送数据.一旦我停止了 influxdb，就没有更多的例外了.

Some other process is sending data to kafka. In my case influxdb was sending data to kafka port because kapacitor also listens on 9092 port. so influxdb is sending data to kafka. Once i stopped influxdb no more exceptions.

这篇关于无法在 CentOS 7 上启动 kafka 代理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！