环境变量PYSPARK_PYTHON和PYSPARK_DRIVER_PYTHON [英] environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON

问题描述

我最近安装了pyspark.它已正确安装.当我在python中使用以下简单程序时，出现错误.

I have installed pyspark recently. It was installed correctly. When I am using following simple program in python, I am getting an error.

>>from pyspark import SparkContext
>>sc = SparkContext()
>>data = range(1,1000)
>>rdd = sc.parallelize(data)
>>rdd.collect()

在运行最后一行时出现错误，其关键行似乎是

while running the last line I am getting error whose key line seems to be

[Stage 0:>                                                          (0 + 0) / 4]18/01/15 14:36:32 ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 123, in main
    ("%d.%d" % sys.version_info[:2], version))
Exception: Python in worker has different version 2.7 than that in driver 3.5, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

我在.bashrc中有以下变量

I have the following variables in .bashrc

export SPARK_HOME=/opt/spark
export PYTHONPATH=$SPARK_HOME/python3

我正在使用Python 3.

I am using Python 3.

推荐答案

顺便说一句，如果您使用PyCharm，则可以添加PYSPARK_PYTHON和PYSPARK_DRIVER_PYTHON来运行/调试下面每个图像的配置

By the way, if you use PyCharm, you could add PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON to run/debug configurations per image below

这篇关于环境变量PYSPARK_PYTHON和PYSPARK_DRIVER_PYTHON的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！