使用Apache Spark提交Python应用程序提交 [英] Submitting Python Application with Apache Spark Submit

问题描述

我正在尝试遵循Apache Spark文档站点上的示例:

I am trying to follow the examples on the Apache Spark documentation site: https://spark.apache.org/docs/2.0.0-preview/submitting-applications.html

我启动了一个Spark独立集群，并希望运行示例Python应用程序.我在我的spark-2.0.0-bin-hadoop2.7目录中，并运行以下命令

I started a Spark standalone cluster and want to run the example Python application. I am in my spark-2.0.0-bin-hadoop2.7 directory and ran the following command

./bin/spark-submit \
--master spark://207.184.161.138:7077 \
examples/src/main/python/pi.py \
1000

但是，我得到了错误

jupyter: '/Users/MyName/spark-2.0.0-bin- \
hadoop2.7/examples/src/main/python/pi.py' is not a Jupyter command

这是我的bash_profile的样子

This is what my bash_profile looks like

#setting path for Spark
export SPARK_PATH=~/spark-2.0.0-bin-hadoop2.7
export PYSPARK_DRIVER_PYTHON="jupyter"
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'

我在做什么错了?

推荐答案

在spark-submit命令之前添加 PYSPARK_DRIVER_PYTHON = ipython .

Add PYSPARK_DRIVER_PYTHON=ipython before the spark-submit command.

示例:

PYSPARK_DRIVER_PYTHON=ipython ./bin/spark-submit \ 
/home/SimpleApp.py

这篇关于使用Apache Spark提交Python应用程序提交的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！