如何在EMR上为Spark/Zeppelin设置spark.driver.memory [英] How to set spark.driver.memory for Spark/Zeppelin on EMR

问题描述

当使用EMR(与Spark，Zeppelin)一起使用时，在Zeppelin Spark解释器设置中更改spark.driver.memory无效.

When using EMR (with Spark, Zeppelin), changing spark.driver.memory in Zeppelin Spark interpreter settings won't work.

我想知道使用EMR Web界面(不是AWS CLI)创建集群时设置Spark驱动程序内存的最佳和最快方法是什么吗?

I wonder what is the best and quickest way to set Spark driver memory when using EMR web interface (not aws CLI) to create clusters?

Bootstrap动作可以解决吗? 如果是，请提供一个示例，说明引导操作文件的外观如何?

Is Bootstrap action could be a solution? If yes, can you please provide an example of how the bootstrap action file should look like?

推荐答案

您始终可以尝试在作业流程/集群创建中添加以下配置:

You can always try to add the following configuration on job flow/cluster creation :

[
    {
        "Classification": "spark-defaults",
        "Properties": {
            "spark.driver.memory": "12G"
        }
    }
]

无论是spark-default，hadoop core-site等，您都可以执行大多数配置.

You can do this most of the configurations whether for spark-default, hadoop core-site, etc.

我希望这会有所帮助！

I hope this helps !

这篇关于如何在EMR上为Spark/Zeppelin设置spark.driver.memory的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！