如何设置星火执行人的记忆？ [英] How to set Spark executor memory?

问题描述

我已经将 spark.executor.memory 为2048米，然后在UI环境页面中，我可以看到这个值已正确设置。但在执行人页面上，我看到这里只有1执行人及其内存的 265.4MB 。很奇怪的值。为什么不256MB，或正如我设置？

I have set spark.executor.memory to 2048m, and in the UI "Environment" page, I can see this value has been set correctly. But in the "Executors" page, I saw there's only 1 executor and its memory is 265.4MB. Very strange value. why not 256MB, or just as what I set?

我是什么在这里失踪？

推荐答案

在用户界面上的执行者标签还包括在列表中的驱动程序。它的执行者ID被列为＆LT;驾驶GT; 。这个过程不是由星火启动，因此它不会受 spark.executor.memory 。

The "Executors" tab on the UI also includes the driver in the list. Its "executor ID" is listed as <driver>. This process is not started by Spark, so it is not affected by spark.executor.memory.

• 如果你开始与司机火花提交由 spark.driver.memory ，其内存最大可控制>或 - 驱动程序内存


• 如果你开始它作为一个普通的旧Java程序，使用普通的 -Xmx Java的标志。

• If you start the driver with spark-submit, its maximal memory can be controlled by spark.driver.memory or --driver-memory
• If you start it as a plain old Java program, use the usual -Xmx Java flag.

这篇关于如何设置星火执行人的记忆？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！