使用spark-submit运行spark时找不到python3.7? [英] cannot find python3.7 when runing spark using spark-submit?

问题描述

有一个困扰我几天的问题.当我使用spark-submit运行Spark应用程序时会发生这种情况，如下所示:

There is a problem that has bothered me for a few days. It occurs when I am running Spark application using spark-submit, it shows like this below:

09:14:36 Exception in thread "main" java.io.IOException: Cannot run program "python3.7": error=2, No such file or directory
09:14:36    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
09:14:36    at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:100)
09:14:36    at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala)
09:14:36    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
09:14:36    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
09:14:36    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
09:14:36    at java.lang.reflect.Method.invoke(Method.java:498)
09:14:36    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
09:14:36    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
09:14:36    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
09:14:36    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
09:14:36    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
09:14:36    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
09:14:36 Caused by: java.io.IOException: error=2, No such file or directory
09:14:36    at java.lang.UNIXProcess.forkAndExec(Native Method)
09:14:36    at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
09:14:36    at java.lang.ProcessImpl.start(ProcessImpl.java:134)
09:14:36    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
09:14:36    ... 12 more

我已经在〜/.bashrc 和/etc/profile 中配置了所有环境变量.而且我绝对可以确保所有环境变量都是正确的.而且我可以在Ubuntu中运行 python3.7 或PySpark.

I have configured all the environment Variables both in ~/.bashrc and /etc/profile. And I am absolutely sure that all the environment variables are correct. And I can run python3.7 or PySpark in my Ubuntu.

export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export PATH=${PATH}:${JAVA_HOME}
export PATH=${PATH}:/mwdata/python3.7/bin
export SPARK_HOME=/srv/spark
export PATH=${PATH}:${SPARK_HOME}/bin
export PYSPARK_PYTHON=python3.7
export PYSPARK_DRIVER_PYTHON=python3.7

那么为什么找不到 python3.7 ?

推荐答案

PySpark的核心取决于Py4J(当前版本为0.10.7)

At its core PySpark depends on Py4J (currently version 0.10.7)

不幸的是，当前的py4j 0.10.7版本与Python 3.7不兼容.

Unfortunately current version of py4j 0.10.7 is not compatible with Python 3.7.

这篇关于使用spark-submit运行spark时找不到python3.7?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！