如何在Spark中正确设置python版本? [英] How to correctly set python version in Spark?

问题描述

我的spark版本是2.4.0，它具有python2.7和python 3.7.默认版本是python2.7.现在我想提交一个使用python3.7的pyspark程序.我尝试了两种方法，但是它们都不起作用.

My spark version is 2.4.0, it has python2.7 and python 3.7 . The default version is python2.7. Now I want to submit a pyspark program which uses python3.7. I tried two ways, but both of them don't work.

spark2-submit --master yarn \ 
--conf "spark.pyspark.python=/usr/bin/python3" \
--conf "spark.pyspark.driver.python=/usr/bin/python3"   pi.py

说不了

Cannot run program "/usr/bin/python3": error=13, Permission denied

但是实际上，我有许可，例如，我可以使用/usr/bin/python3 test.py 运行python程序.

But actually, I have the permission, for example, I can use /usr/bin/python3 test.py to run a python program.

export PYSPARK_PYTHON=/usr/bin/python3
export PYSPARK_DRIVER_PYTHON=/usr/bin/python3

通过这种方式，spark根本无法使用python3.

In this way, spark can't use python3 at all.

推荐答案

我遇到了同样的问题.

在脚本开头配置环境的解决方案(在 Spark无法执行任务中)对我不起作用.

Solution of configuring the env in the beginning of the script (in Spark not executing tasks) did not work for me.

无需重新启动集群，只需执行以下命令即可.

Without restarting the cluster, just executing the command below worked for me.

sudo sed -i -e '$a\export PYSPARK_PYTHON=/usr/bin/python3' /etc/spark/conf/spark-env.sh

这篇关于如何在Spark中正确设置python版本?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！