了解Spark Shuffle溢出 [英] Understanding Spark shuffle spill
问题描述
如果我理解正确，那么当reduce任务开始收集其输入的shuffle块(来自不同map任务的输出)时，它将首先将它们保存在内存中(Q1).当执行器的重新混合保留的内存量用尽时(在更改内存管理(Q2)之前)，内存中的数据将堆积"到磁盘上.如果spark.shuffle.spill.compress为true，则以压缩方式将内存中的数据写入磁盘.
If I understand correctly, when a reduce task goes about gathering its input shuffle blocks ( from outputs of different map tasks ) it first keeps them in memory ( Q1 ). When the amount of shuffles-reserved memory of an executor ( before the change in memory management ( Q2 ) ) is exhausted, the in-memory data is "spilled" to disk. if spark.shuffle.spill.compress is true then that in-memory data is written to disk in a compressed fashion.
我的问题:
Q0:我的理解正确吗?
Q0: Is my understanding correct?
问题1:reduce任务中收集的数据是否始终未压缩?
Q1: Is the gathered data inside the reduce task always uncompressed?
Q2:如何估算可用于收集随机播放块的执行器内存量?
Q2: How can I estimate the amount of executor memory available for gathering shuffle blocks?
Q3:我已经看到过这样的说法:当数据集无法容纳在内存中时，就会发生随机溢出"，但据我所知，只要保留随机执行器的执行器内存足以容纳所有(未压缩的)随机输入块所有活动任务中，那么就不会发生溢出，对吗?
Q3: I've seen the claim "shuffle spill happens when your dataset cannot fit in memory", but to my understanding as long as the shuffle-reserved executor memory is big enough to contain all the ( uncompressed ) shuffle input blocks of all its ACTIVE tasks, then no spill should occur, is that correct?
如果是这样，为了避免溢出，需要确保在所有并行的reduce端任务中结束的(未压缩的)数据小于执行者的随机保留的内存部分?
If so, to avoid spills one needs to make sure that the ( uncompressed ) data which ends up in all parallel reduce-side tasks is less than the executor's shuffle-reserved memory part?
推荐答案
1.6之前和之后的内存管理有所不同.在这两种情况下，都有执行内存和存储内存的概念.区别在于1.6之前是静态的.意味着有一个配置参数，用于指定执行和存储的内存量.当其中任何一个都不足够时，就会发生溢出.
There are differences in memory management in before and after 1.6. In both cases, there are notions of execution memory and storage memory. The difference is that before 1.6 it's static. Meaning there is a configuration parameter that specifies how much memory is for execution and for storage. And there is a spill, when either one is not enough.
Apache Spark必须解决的问题之一是并发执行:
One of the issues that Apache Spark has to workaround is a concurrent execution of:
- 并行执行的不同阶段
- 不同的任务，例如汇总或排序.
-
我会说你的理解是正确的.
I'd say that your understanding is correct.
内存中的内容未压缩，否则无法处理.执行内存以块的形式溢出到磁盘中，并且可以压缩.
What's in memory is uncompressed or else it cannot be processed. Execution memory is spilled to disk in blocks and as you mentioned can be compressed.
好吧，从1.3.1开始，您可以对其进行配置，然后就知道大小了.至于任何时候剩余的内容，您都可以通过使用类似jstat -gcutil <pid> <period>的执行器进程来查看.它可能会为您提供那里有多少可用内存的线索.知道为存储和执行配置了多少内存，尽可能少的default.parallelism可能会为您提供线索.
Well, since 1.3.1 you can configure it, then you know the size. As of what's left at any moment in time, you can see that by looking at the executor process with something like jstat -gcutil <pid> <period>. It might give you a clue of how much memory is free there. Knowing how much memory is configured for storage and execution, having as little default.parallelism as possible might give you a clue.
是的，但是很难推理.数据中可能存在偏差，例如某些键的值比其他键更多，并行执行的次数很多，等等.
That's true, but it's hard to reason about; there might be skew in the data such as some keys have more values than the others, there are many parallel executions, etc.
这篇关于了解Spark Shuffle溢出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
