Web UI 如何计算存储内存(在 Executors 选项卡中)? [英] How does web UI calculate Storage Memory (in Executors tab)?

问题描述

我正在尝试了解 Spark 2.1.0 如何在节点上分配内存.

I'm trying to understand how Spark 2.1.0 allocates memory on nodes.

假设我正在启动一个本地 PySpark REPL，为其分配 2GB 内存:

Suppose I'm starting a local PySpark REPL assigning it 2GB of memory:

$ pyspark --conf spark.driver.memory=2g

Spark UI 告诉有 956.6 MB 分配给存储内存:

Spark UI tells that there are 956.6 MB allocated for storage memory:

我不明白如何得到那个数字，这是我的思考过程:

I don't understand how to get to that number, this is my thinking process:

1. 驱动程序堆大小设置为2048 MB，
2. 根据文档:(2048 MB- 300 MB) * 0.6 = 1048.8 MB 用于执行和存储区域(统一)，
3. 另外在统一区域内的1048.8 MB * 0.5 = 524.4 MB应保留为免疫存储区域

1. Driver heap size is set to 2048 MB,
2. According to docs: (2048 MB - 300 MB) * 0.6 = 1048.8 MB are used for both execution and storage regions (unified),
3. Additionally 1048.8 MB * 0.5 = 524.4 MB within unified region should be reserved as immune storage region

那么，Spark 中 956.6 MB 的实际值是如何计算的?

推荐答案

您似乎在使用 local 模式(一个驱动程序也充当唯一的执行程序)，但它也应该适用到其他集群模式.

You seem to be using local mode (with one driver that also acts as the only executor), but it should also be applicable to other clustered modes.

为 BlockManagerMasterEndpoint 启用 INFO 日志记录级别，以了解 Spark 看到您在命令行上设置的属性(如 spark.driver.memory)有多少内存.

Enable the INFO logging level for BlockManagerMasterEndpoint to know how much memory Spark sees the property you set on the command line (as spark.driver.memory).

log4j.logger.org.apache.spark.storage.BlockManagerMasterEndpoint=INFO

当您启动 spark-shell --conf spark.driver.memory=2g 时，您将看到以下内容:

When you start spark-shell --conf spark.driver.memory=2g you'll see the following:

$ ./bin/spark-shell --conf spark.driver.memory=2g
...
17/05/07 15:20:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.8:57177 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8, 57177, None)

如您所见，可用内存为 912.3，计算如下(请参阅 UnifiedMemoryManager.getMaxMemory):

As you can see the available memory is 912.3 which is calculated as follows (see UnifiedMemoryManager.getMaxMemory):

// local mode with --conf spark.driver.memory=2g
scala> sc.getConf.getSizeAsBytes("spark.driver.memory")
res0: Long = 2147483648

scala> val systemMemory = Runtime.getRuntime.maxMemory

// fixed amount of memory for non-storage, non-execution purposes
val reservedMemory = 300 * 1024 * 1024

// minimum system memory required
val minSystemMemory = (reservedMemory * 1.5).ceil.toLong

val usableMemory = systemMemory - reservedMemory

val memoryFraction = sc.getConf.getDouble("spark.memory.fraction", 0.6)
scala> val maxMemory = (usableMemory * memoryFraction).toLong
maxMemory: Long = 956615884

import org.apache.spark.network.util.JavaUtils
scala> JavaUtils.byteStringAsMb(maxMemory + "b")
res1: Long = 912

让我们回顾一下 Web UI 是如何计算内存的(这与上面的不同，应该只是显示值！).这就是令人惊讶的部分.

Let's review how web UI calculates the memory (which is different from what's above and is supposed to just display the value!). That's the surprising part.

存储内存在 Web UI 中的显示方式由 utils.js(映射到 Scala)如下所示:

How the Storage Memory is displayed in web UI is controlled by the custom JavaScript function formatBytes in utils.js that (mapped to Scala) looks as follows:

def formatBytes(bytes: Double) = {
  val k = 1000
  val i = math.floor(math.log(bytes) / math.log(k))
  val maxMemoryWebUI = bytes / math.pow(k, i)
  f"$maxMemoryWebUI%1.1f"
}
scala> println(formatBytes(maxMemory))
956.6

956.6！这正是 Web UI 显示的内容，与 Spark 的 UnifiedMemoryManager 考虑的可用内存完全不同.非常令人惊讶，不是吗?

956.6! That's exactly what web UI shows and is quite different from what Spark's UnifiedMemoryManager considers the available memory. Quite surprising, isn't it?

我认为这是一个错误并将其填充为 SPARK-20691.

I think it's a bug and filled it as SPARK-20691.

这篇关于Web UI 如何计算存储内存(在 Executors 选项卡中)?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！