java.lang.IllegalArgumentException kafka 控制台消费者 [英] java.lang.IllegalArgumentException kafka console consumer

问题描述

我们在我们的环境中使用 Kafka 2.10-0.9.0.2.4.2.0-258.我们在几个主题上遇到了 kafka 控制台消费者的异常.我知道有时进入这些主题的消息太大，但它们不会超过 message.max.bytes.

We are using Kafka 2.10-0.9.0.2.4.2.0-258 in our environments. We are getting below exception with kafka console consumer on few topics. I am aware that some times messages coming into these topics are too big but they do not exceed message.max.bytes.

./kafka-console-consumer.sh --zookeeper xxx:2181,xxx:2181,xxx:2181 --topic test-topic

{metadata.broker.list=xxx:9092,xxx:9092,xxx:9092, request.timeout.ms=30000, client.id=console-consumer-76015, security.protocol=PLAINTEXT}
[2016-08-28 21:27:54,795] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)

java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:275)
        at kafka.message.Message.sliceDelimited(Message.scala:237)
        at kafka.message.Message.key(Message.scala:224)
        at kafka.message.MessageAndMetadata.key(MessageAndMetadata.scala:30)
        at kafka.consumer.OldConsumer.receive(BaseConsumer.scala:84)
        at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:109)
        at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:69)
        at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:47)
        at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Processed a total of 0 messages

我按照以下链接的建议将 replica.fetch.max.bytes 减少到等于 message.max.bytes 和 num.replica.fetchers 到 2，但没有解决.

I decreased replica.fetch.max.bytes equal to message.max.bytes and also num.replica.fetchers to 2 as suggested in link below but it did not resolve.

https://issues.apache.org/jira/browse/KAFKA-1196

知道我还应该做些什么来使它工作吗?

Any idea what else I should do to make it work?

任何帮助将不胜感激.

提前致谢.

推荐答案

我遇到了完全相同的问题.根本原因是您的 kafka 安装使用的 kafka jar 文件与您用于开发和运行生产者的 jar 文件不兼容.您可以在/usr/hdp/current/kafka-broker/libs 中找到您的安装使用的 kafka jar 版本

I was having the exact same issue. The root cause was the incompatibility of the kafka jar file between the one that your kafka installation use and the one that you used to develop and run your producer. You can find which version of kafka jars your installation is using in /usr/hdp/current/kafka-broker/libs

就我而言，我的 kafka 安装使用的是 kafka_2.10-0.9.0.2.4.2.0-258.jar，但我与生产者捆绑的 kafka jar 是 0.10.0.1.有一次，我切换到 0.9.0.2.4.2.0-258，它起作用了.

In my case, my kafka installation is using kafka_2.10-0.9.0.2.4.2.0-258.jar, but the kafka jar that I bundled with my producer was 0.10.0.1. Once, I switched to 0.9.0.2.4.2.0-258, it worked.

如果您的集群是 HDP 并且您正在使用 maven 构建您的生产者，您可以在此处找到所有 jar 依赖项 http://repo.hortonworks.com/content/repositories/releases/

If your cluster is HDP and you are using maven to build your producer, you can find all jar dependencies here http://repo.hortonworks.com/content/repositories/releases/

对于 maven，这里是你必须使用的:

For maven here is what you have to use:

存储库:

<repositories>
    <repository>
        <id>org.hortonworks</id>
        <url>http://repo.hortonworks.com/content/repositories/releases/</url>
    </repository>
</repositories>

依赖:

       <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.10</artifactId>
            <version>0.9.0.2.4.2.0-258</version>
            <scope>compile</scope>
            <exclusions>
                <exclusion>
                    <artifactId>jmxri</artifactId>
                    <groupId>com.sun.jmx</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jms</artifactId>
                    <groupId>javax.jms</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>jmxtools</artifactId>
                    <groupId>com.sun.jdmk</groupId>
                </exclusion>
            </exclusions>
        </dependency>

这篇关于java.lang.IllegalArgumentException kafka 控制台消费者的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！