如何使用YARN客户端模式在Jupyter中交互运行Pyspark? [英] How can I run Pyspark interactively in Jupyter using YARN-client mode?

问题描述

现在，我已经通过本

Now I have succeeded in running Pyspark in Jupyter in local mode by the second method as mentioned in this blog. Here is the code:

import findspark
findspark.init()
from pyspark import SparkContext
sc = SparkContext("local", "First App")

我想在YARN客户端模式下交互式运行它，该怎么办? 让我们进一步，如何在独立模式和YARN群集模式下运行.

I want to run it interactively in YARN-client mode,how can I do it? Let's go futher,how to run in different modes,e.g.standalone mode and YARN-cluster mode.

推荐答案

根据

主URL接受基于HADOOP_CONF_DIR或YARN_CONF_DIR变量的yarn参数

所以我可以简单地使用: sc = SparkContext("yarn-client", "First App")

So I can simply use: sc = SparkContext("yarn-client", "First App")

这篇关于如何使用YARN客户端模式在Jupyter中交互运行Pyspark?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！