Spark Streaming Kafka 集成直接方法 EOFException [英] Spark Streaming Kafka Integration direct Approach EOFException

问题描述

当我运行火花流示例 org.apache.spark.examples.streaming.JavaDirectKafkaWordCount 时，我发现了一个 EOFException 跟随，我该如何解决它

when i run spark streaming example org.apache.spark.examples.streaming.JavaDirectKafkaWordCount,i caught an EOFException follow,how can I resolve it

Exception in thread "main" org.apache.spark.SparkException: java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
    at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
    at org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
    at scala.util.Either.fold(Either.scala:97)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412)
    at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528)
    at org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)

推荐答案

直接流使用底层的 Kafka 消费者，因此需要提供代理列表.很可能您还没有，因此您可能希望以 bhost1:9092,bhost2:9092,...,bhostN:9092 形式设置 metadata.broker.list 属性>.

The direct stream uses a low level Kafka consumer underneath therefore one needs to provide the list of brokers. Most likely you haven't so you may want to set the metadata.broker.list property in the form bhost1:9092,bhost2:9092,...,bhostN:9092.

另见 Kafka SimpleConsumer 无法连接到zookeeper:从频道读取时收到 -1.

这篇关于Spark Streaming Kafka 集成直接方法 EOFException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！