如何知道PySpark应用程序的部署模式? [英] How to know deploy mode of PySpark application?

问题描述

我正在尝试解决内存不足的问题，并且我想知道是否需要在spark主文件夹的默认配置文件(spark-defaults.conf)中更改这些设置.或者，如果我可以在代码中进行设置.

I am trying to fix an issue with running out of memory, and I want to know whether I need to change these settings in the default configurations file (spark-defaults.conf) in the spark home folder. Or, if I can set them in the code.

我看到了这个问题 PySpark:java.lang.OutofMemoryError:Java堆空间，它取决于我是否以client模式运行.我正在集群上运行spark并使用独立监视它.

I saw this question PySpark: java.lang.OutofMemoryError: Java heap space and it says that it depends on if I'm running in client mode. I'm running spark on a cluster and monitoring it using standalone.

但是，如何确定我是否在client模式下运行spark?

But, how do I figure out if I'm running spark in client mode?

推荐答案

由于sc.deployMode在PySpark中不可用，因此您可以检查spark.submit.deployMode

Since sc.deployMode is not available in PySpark, you could check spark.submit.deployMode

scala> sc.getConf.get("spark.submit.deployMode")
res0: String = client

在PySpark中不可用

使用sc.deployMode

scala> sc.deployMode
res0: String = client

scala> sc.version
res1: String = 2.1.0-SNAPSHOT

这篇关于如何知道PySpark应用程序的部署模式?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！