错误:无法找到 py4j,您的 SPARK_HOME 可能没有正确配置 [英] ERROR: Unable to find py4j, your SPARK_HOME may not be configured correctly
本文介绍了错误:无法找到 py4j,您的 SPARK_HOME 可能没有正确配置的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我无法在 Jupyter 笔记本中运行以下导入.
I'm unable to run below import in Jupyter notebook.
findspark.init('home/ubuntu/spark-3.0.0-bin-hadoop3.2')
出现以下错误:
---------------------------------------------------------------------------
~/.local/lib/python3.6/site-packages/findspark.py in init(spark_home, python_path, edit_rc, edit_profile)
144 except IndexError:
145 raise Exception(
--> 146 "Unable to find py4j, your SPARK_HOME may not be configured correctly"
147 )
148 sys.path[:0] = [spark_python, py4j]
Exception: Unable to find py4j, your SPARK_HOME may not be configured correctly
我确实安装了 py4j 并且还尝试将以下几行添加到 ~/.bashrc
I do have py4j installed and also tried to add these below lines into ~/.bashrc
export SPARK_HOME=/home/ubuntu/spark-3.0.0-bin-hadoop3.2
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.10.9-src.zip:$PYTHONPATH
推荐答案
检查你安装的spark版本是否和你在SPARK_HOME名下声明的一样
Check if the spark version you installed is the same that you declare under SPARK_HOME name
例如(在 Google Colab 中),我已经安装:
For example (in Google Colab), I've installed:
!wget -q https://downloads.apache.org/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz
然后我声明:
os.environ["SPARK_HOME"] = "/content/spark-3.0.1-bin-hadoop3.2"
看看 spark-3.0.1-bin-hadoop3.2 在两个地方必须相同
Look that spark-3.0.1-bin-hadoop3.2 must be same in both places
这篇关于错误:无法找到 py4j,您的 SPARK_HOME 可能没有正确配置的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文