初始化SparkContext时jvm错误中不存在pyspark错误 [英] pyspark error does not exist in the jvm error when initializing SparkContext

问题描述

我在emr上使用spark并编写了pyspark脚本， 尝试

I am using spark over emr and writing a pyspark script, I am getting an error when trying to

from pyspark import SparkContext
sc = SparkContext()

这是错误

File "pyex.py", line 5, in <module>
    sc = SparkContext()   File "/usr/local/lib/python3.4/site-packages/pyspark/context.py", line 118, in __init__
    conf, jsc, profiler_cls)   File "/usr/local/lib/python3.4/site-packages/pyspark/context.py", line 195, in _do_init
    self._encryption_enabled = self._jvm.PythonUtils.getEncryptionEnabled(self._jsc)   File "/usr/local/lib/python3.4/site-packages/py4j/java_gateway.py", line 1487, in __getattr__
    "{0}.{1} does not exist in the JVM".format(self._fqn, name)) py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getEncryptionEnabled does not exist in the JVM

我找到了此答案，指出我需要导入sparkcontext，但这也不起作用.

I found this answer stating that I need to import sparkcontext but this is not working also.

推荐答案

PySpark最近发布了2.4.0，但是尚没有稳定的版本可以与此新版本同时出现.尝试降级到pyspark 2.3.2，这对我来说已经解决了

PySpark recently released 2.4.0, but there's no stable release for spark coinciding with this new version. Try downgrading to pyspark 2.3.2, this fixed it for me

为更清楚起见，您的PySpark版本必须与下载的Apache Spark版本相同，否则您可能会遇到兼容性问题

to be more clear your PySpark version needs to be the same as the Apache Spark version that is downloaded, or you may run into compatibility issues

使用

点冻结

这篇关于初始化SparkContext时jvm错误中不存在pyspark错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！