将PySpark与Jupyter Notebook集成 [英] Integrate PySpark with Jupyter Notebook
问题描述
我正在关注网站以安装Jupyter Notebook,PySpark和整合两者。
I'm following this site to install Jupyter Notebook, PySpark and integrate both.
当我需要创建Jupyter配置文件时,我读到Jupyter配置文件不再存在。所以我继续执行以下几行。
When I needed to create the "Jupyter profile", I read that "Jupyter profiles" not longer exist. So I continue executing the following lines.
$ mkdir -p ~/.ipython/kernels/pyspark
$ touch ~/.ipython/kernels/pyspark/kernel.json
我打开 kernel.json
并写下以下内容:
I opened kernel.json
and write the following:
{
"display_name": "pySpark",
"language": "python",
"argv": [
"/usr/bin/python",
"-m",
"IPython.kernel",
"-f",
"{connection_file}"
],
"env": {
"SPARK_HOME": "/usr/local/Cellar/spark-2.0.0-bin-hadoop2.7",
"PYTHONPATH": "/usr/local/Cellar/spark-2.0.0-bin-hadoop2.7/python:/usr/local/Cellar/spark-2.0.0-bin-hadoop2.7/python/lib/py4j-0.10.1-src.zip",
"PYTHONSTARTUP": "/usr/local/Cellar/spark-2.0.0-bin-hadoop2.7/python/pyspark/shell.py",
"PYSPARK_SUBMIT_ARGS": "pyspark-shell"
}
}
Spark的路径是正确的。
The paths of Spark are correct.
但是,当我运行 jupyter console --ke rnel pyspark
我得到了这个输出:
But then, when I run jupyter console --kernel pyspark
I get this output:
MacBook:~ Agus$ jupyter console --kernel pyspark
/usr/bin/python: No module named IPython
Traceback (most recent call last):
File "/usr/local/bin/jupyter-console", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/jupyter_core/application.py", line 267, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/usr/local/lib/python2.7/site-packages/traitlets/config/application.py", line 595, in launch_instance
app.initialize(argv)
File "<decorator-gen-113>", line 2, in initialize
File "/usr/local/lib/python2.7/site-packages/traitlets/config/application.py", line 74, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/jupyter_console/app.py", line 137, in initialize
self.init_shell()
File "/usr/local/lib/python2.7/site-packages/jupyter_console/app.py", line 110, in init_shell
client=self.kernel_client,
File "/usr/local/lib/python2.7/site-packages/traitlets/config/configurable.py", line 412, in instance
inst = cls(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/jupyter_console/ptshell.py", line 251, in __init__
self.init_kernel_info()
File "/usr/local/lib/python2.7/site-packages/jupyter_console/ptshell.py", line 305, in init_kernel_info
raise RuntimeError("Kernel didn't respond to kernel_info_request")
RuntimeError: Kernel didn't respond to kernel_info_request
推荐答案
将pyspark与jupyter笔记本集成的许多方法。< br>
1. 安装 Apache Toree 。
Many ways to integrate pyspark with jupyter notebook.
1.Install Apache Toree.
pip install jupyter
pip install toree
jupyter toree install --spark_home=path/to/your/spark_directory --interpreters=PySpark
您可以通过
jupyter kernelspec list
你会得到一个条目toree pyspark内核
you will get an entry for toree pyspark kernel
apache_toree_pyspark /home/pauli/.local/share/jupyter/kernels/apache_toree_pyspark
之后如果需要,你可以安装其他的解释器,如SparkR,Scala,SQL
Afterwards if you want, you can install other intepreters like SparkR,Scala,SQL
jupyter toree install --interpreters=Scala,SparkR,SQL
2. 将这些行添加到bashrc
export SPARK_HOME=/path to /spark-2.2.0
export PATH="$PATH:$SPARK_HOME/bin"
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
在终端输入 pyspark
它将打开一个初始化了sparkcontext的jupyter笔记本。
type pyspark
in terminal and it will open a jupyter notebook with sparkcontext initialized.
-
安装
pyspark
仅作为 python包
pip install pyspark
现在您可以像其他python包一样导入pyspark。
Now you can import pyspark like another python package.
这篇关于将PySpark与Jupyter Notebook集成的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!