如何设置火花驱动程序的Python版本？ [英] How do I set the driver's python version in spark?

问题描述

我使用的火花1.4.0-RC2这样我就可以使用Python 3火花。如果我添加出口PYSPARK_PYTHON = python3 我的.bashrc文件，我可以交互使用Python 3，运行火花但是，如果我想在本地模式下运行一个独立的程序，我得到一个错误：

I'm using spark 1.4.0-rc2 so I can use python 3 with spark. If I add export PYSPARK_PYTHON=python3 to my .bashrc file, I can run spark interactively with python 3. However, if I want to run a standalone program in local mode, I get an error:

Exception: Python in worker has different version 3.4 than that in driver 2.7, PySpark cannot run with different minor versions

我如何指定驱动程序的Python版本？设置出口PYSPARK_DRIVER_PYTHON = python3 没有工作。

推荐答案

您需要确保你推出如果通过提交独立程序启动与蟒蛇3.独立的项目火花提交那么它应该工作很好，但如果你正在使用Python启动它确保您使用python3启动您的应用程序。

You need to make sure the standalone project you're launching is launched with python 3. If your are submitting your standalone program through spark-submit then it should work fine, but if you are launching it with python make sure you use python3 to start your app.

另外，还要确保你在 ./的conf / spark-spek-env.sh （如果存在的犯规你可以使用它火花env.sh.template 的位置。

Also make sure you have set your env variables in ./conf/spark-spek-env.sh (if it doesnt exist you can use spark-env.sh.template as a base.

这篇关于如何设置火花驱动程序的Python版本？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！