Spark Shell依赖异常 [英] spark shell dependency exception

问题描述

我的主机系统Windows 10上有cloudera vm，而我的spark版本是1.6.我试图使用以下命令加载spark-shell.

My host system windows 10 and I have cloudera vm and my spark version is 1.6 .I was trying to load the spark-shell with the below command.

spark-shell --packages org.apache.spark:spark-streaming-twitter_2.10:1.6.0

但是它抛出了以下异常:

:::: ERRORS Server access error at url https://repo1.maven.org/maven2/org/apache/spark/spark-streaming-twitter_2.10/1.6.0/spark-streaming-twitter_2.10-1.6.0.pom (javax.net.ssl.SSLException: Received fatal alert: protocol_version)
    Server access error at url https://repo1.maven.org/maven2/org/apache/spark/spark-streaming-twitter_2.10/1.6.0/spark-streaming-twitter_2.10-1.6.0.jar (javax.net.ssl.SSLException: Received fatal alert: protocol_version)

::使用细节或调试消息级别获取更多详细信息线程

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS Exception in thread

"main" java.lang.RuntimeException: [unresolved dependency: org.apache.spark#spark-streaming-twitter_2.10;1.6.0: not found] at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1067) at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:287) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:154) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

推荐答案

我查看了:因此，它似乎与Java TLS协议版本默认值相关.如果Java版本已过时，并且导致HTTPS上的请求的TLS版本已过时，则它将被阻止.我在尝试安装PySpark软件包时遇到了这个问题.现在，许多服务器都阻止了TLS的过时版本.例如，请参见: Github现在可以阻止较弱的加密标准

So it appears that it is related to the Java TLS protocol version defaults. If the Java version is outdated and results in an outdated TLS version for the request over HTTPS, then it gets blocked. I ran into this issue while trying to install a PySpark package. Many servers are now blocking outdated versions of TLS. See, for example: Github now blocks weak cryptography standards

解决方案是使用环境变量强制使用TLS版本1.2:

The solution was to force TLS version 1.2 with an environment variable:

echo 'export JAVA_TOOL_OPTIONS="-Dhttps.protocols=TLSv1.2"' >> ~/.bashrc
source ~/.bashrc

当我重新运行命令以用我的软件包启动PySpark时:

When I re-ran the command to start PySpark with my package:

pyspark --packages com.databricks:spark-csv_2.10:1.5.0

我指定的TLS版本立即被检测到.从字面上看，它给了我输出:

The TLS version that I specified was detected immediately. It literally gave me the output:

拾取了JAVA_TOOL_OPTIONS:-Dhttps.protocols = TLSv1.2

Picked up JAVA_TOOL_OPTIONS: -Dhttps.protocols=TLSv1.2

这篇关于Spark Shell依赖异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！