在Apache Spark和Kafka代理之间启用SSL [英] Enabling SSL between Apache spark and Kafka broker

问题描述

我正在尝试在Apache Spark 1.4.1和Kafka 0.9.0.0之间启用SSL，并且我正在使用 spark-streaming-kafka_2.10 Jar连接到Kafka，并且正在使用 KafkaUtils.createDirectStream 方法从Kafka主题中读取数据.

I am trying to enable the SSL between my Apache Spark 1.4.1 and Kafka 0.9.0.0 and I am using spark-streaming-kafka_2.10 Jar to connect to Kafka and I am using KafkaUtils.createDirectStream method to read the data from Kafka topic.

最初，我遇到了OOM问题，并通过增加驱动程序内存来​​解决它，此后，我看到下面的问题，我做了一点阅读，发现 spark-streaming-kafka_2.10 使用Kafka 0.8.2.1 API，该API不支持SSL(Kafka仅在0.9.0.0版本之后才支持SSL).在Spark 1.4.1和Kafka 0.9.0.0之间有启用SSL的替代方法.

Initially, I got OOM issue and I have resolved it by increasing the Driver memory, after that I am seeing below issue, I have done little bit of reading and found out that spark-streaming-kafka_2.10 uses Kafka 0.8.2.1 API, which doesn't support SSL (Kafka supports SSL only after 0.9.0.0 versions). are there any alternatives to enable SSL between Spark 1.4.1 and Kafka 0.9.0.0.

• http://grokbase.com/p/kafka/users/158wy0wtxk/ssl-between-kafka-and-spark-streaming-api
• Kafka和Spark之间的SSL
• http://spark.apache.org/docs/latest/streaming-kafka-integration.html
• https://issues.apache.org/jira/browse/SPARK-15089

这是日志

iableProperties: Property security.protocol is not valid
16/10/24 18:25:09 WARN utils.VerifiableProperties: Property ssl.truststore.location is not valid
16/10/24 18:25:09 WARN utils.VerifiableProperties: Property ssl.truststore.password is not valid
16/10/24 18:25:09 INFO utils.VerifiableProperties: Property zookeeper.connect is overridden to 
16/10/24 18:25:09 INFO consumer.SimpleConsumer: Reconnect due to error

Exception in thread "main" org.apache.spark.SparkException: java.io.EOFException
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:99)

推荐答案

Spark 2.0已添加了对Kafka 0.10(具有SSL支持)的支持.我们必须使用Maven工件spark-streaming-kafka-0-10_2.10.

Support for Kafka 0.10 (which has SSL support) has been added to Spark 2.0. we have to use maven artifact spark-streaming-kafka-0-10_2.10.

这篇关于在Apache Spark和Kafka代理之间启用SSL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！