如何更新在 Mac 上的 PyCharm 中运行的 pyspark 使用的 Java 密钥库? [英] How do I update the Java keystore used by pyspark running in PyCharm on Mac?

问题描述

我在 Mac 上运行 PyCharm 2018.2 并执行 pyspark 程序.Spark 安装在 virtualenv 中.

I'm running PyCharm 2018.2 on a Mac and executing a pyspark program. Spark was installed in the virtualenv.

我需要在我的 pyspark 脚本中使用外部 jars(特别是 AWS s3 jars)，所以我使用以下内容来声明 maven 依赖:

I need to use external jars (specifically, the AWS s3 jars) in my pyspark script, so I use the following to declare the maven dependancy:

import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages "org.apache.hadoop:hadoop-aws:2.7.3" pyspark-shell'
conf = SparkConf() \
    .setMaster("local[2]") \
    .setAppName("pyspark-unittests") \
    .set("spark.sql.parquet.compression.codec", "snappy")

sc = SparkContext(conf=conf)

在我的家庭网络上，这很好用.

On my home network, this works great.

在我的公司网络中，我和互联网之间有一个 SSL 检查器，用于将 HTTPS 请求中的 SSL 证书交换到 Maven Central.

On my corporate network, there is an SSL inspector between me and the internet that swaps the SSL certificates on the HTTPS requests to Maven Central.

这会导致以下错误消息:

This results in the below error message:

Server access error at url https://repo1.maven.org/maven2/joda-time/joda-time/maven-metadata.xml (javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target)

我知道这是因为 pyspark 执行的 JVM 不信任 SSL 检查器用于签署 HTTPS 响应的 SSL 证书.

I know this is because the SSL certificates that the SSL inspector signed the HTTPS response with are not trusted by the JVM executed by pyspark.

我有签署 HTTPS 响应的中间证书的 .cer 副本.

I have .cer copies of the intermediate certificates that are signing the HTTPS response.

在这种特定情况下使用了哪个 JVM(在 PyCharm 中运行的 python/pyspark)，以及如何更新该 JVM 信任库中的证书?

Which JVM is being used in this specific case (python/pyspark running in PyCharm) and how can I update the certificates in that JVM's trust store?

推荐答案

在调试器中运行程序并在 sc = SparkContext(conf = conf) 后暂停，然后运行 ​​ps aux|grep java 你就会知道哪个 jvm 用于这个

Run the program in a debugger and pause it after sc = SparkContext(conf = conf) and then run ps aux | grep java and you will know which jvm is getting used for this

那么你应该设置相同的密钥库.知道正确的 JDK 后，使用以下链接设置 keyStore

Then you should setup the keystore for the same. After you know the correct JDK, use below link to setup the keyStore

如何正确将自签名证书导入所有 Java 应用程序默认可用的 Java 密钥库?

这篇关于如何更新在 Mac 上的 PyCharm 中运行的 pyspark 使用的 Java 密钥库?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！