Pyspark/Pyspark内核在Jupyter Notebook中不起作用 [英] Pyspark / pyspark kernels not working in jupyter notebook

问题描述

以下是已安装的内核:

 $jupyter-kernelspec list


Available kernels:
  apache_toree_scala    /usr/local/share/jupyter/kernels/apache_toree_scala
  apache_toree_sql      /usr/local/share/jupyter/kernels/apache_toree_sql
  pyspark3kernel        /usr/local/share/jupyter/kernels/pyspark3kernel
  pysparkkernel         /usr/local/share/jupyter/kernels/pysparkkernel
  python3               /usr/local/share/jupyter/kernels/python3
  sparkkernel           /usr/local/share/jupyter/kernels/sparkkernel
  sparkrkernel          /usr/local/share/jupyter/kernels/sparkrkernel

已创建一个新笔记本，但失败

The code failed because of a fatal error:
    Error sending http request and maximum retry encountered..

jupyter控制台中没有[错误]消息

解决方案

如果使用magicspark连接Jupiter笔记本，则还应该启动Livy，这是magicspark用于与Spark集群通信的API服务.

1. 从 Apache Livy 下载Livy并将其解压缩
2. 检查是否设置了SPARK_HOME环境，如果未设置，请设置为您的Spark安装目录
3. 在外壳程序/命令行中通过<livy_home>/bin/livy-server运行Livy服务器

现在回到笔记本，您应该能够在单元格中运行spark代码.

Here are installed kernels:

 $jupyter-kernelspec list


Available kernels:
  apache_toree_scala    /usr/local/share/jupyter/kernels/apache_toree_scala
  apache_toree_sql      /usr/local/share/jupyter/kernels/apache_toree_sql
  pyspark3kernel        /usr/local/share/jupyter/kernels/pyspark3kernel
  pysparkkernel         /usr/local/share/jupyter/kernels/pysparkkernel
  python3               /usr/local/share/jupyter/kernels/python3
  sparkkernel           /usr/local/share/jupyter/kernels/sparkkernel
  sparkrkernel          /usr/local/share/jupyter/kernels/sparkrkernel

A new notebook was created but fails with

The code failed because of a fatal error:
    Error sending http request and maximum retry encountered..

There is no [error] message in the jupyter console

解决方案

If you use magicspark to connect your Jupiter notebook, you should also start Livy which is API service used by magicspark to talk to your Spark cluster.

1. Download Livy from Apache Livy and unzip it
2. Check SPARK_HOME environment is set, if not, set to your Spark installation directory
3. Run Livy server by <livy_home>/bin/livy-server in the shell/command line

Now go back to your notebook, you should be able to run spark code in cell.

这篇关于Pyspark/Pyspark内核在Jupyter Notebook中不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！