为什么Spark作业会因“打开的文件太多"而失败? [英] Why does Spark job fail with "too many open files"?

问题描述

在我的Spark作业的洗牌阶段，我得到打开的文件太多".为什么我的工作打开这么多文件?我可以采取什么步骤来使我的工作取得成功.

I get "too many open files" during the shuffle phase of my Spark job. Why is my job opening so many files? What steps can I take to try to make my job succeed.

推荐答案

最好的方法肯定是尽可能增加ulimit， 这是我们在Spark中做出的一个假设，即集群将是 能够移动它.

The best way is definitely just to increase the ulimit if possible, this is sort of an assumption we make in Spark that clusters will be able to move it around.

您也许可以通过减少 减速器(或每个节点使用的核心)，但这可能会对您的性能产​​生一些影响 工作.

You might be able to hack around this by decreasing the number of reducers [or cores used by each node] but this could have some performance implications for your job.

通常，如果集群中的节点具有C分配的核心，并且您运行 使用X减速器的作业，Spark将并行打开C * X文件，然后 动笔.洗牌合并将有助于减少总数 创建的文件数，但在任何位置打开的文件句柄数 时间不会改变，因此不会解决ulimit问题.

In general if a node in your cluster has C assigned cores and you run a job with X reducers then Spark will open C*X files in parallel and start writing. Shuffle consolidation will help decrease the total number of files created but the number of file handles open at any time doesn't change so it won't help the ulimit problem.

-帕特里克·温德尔

这篇关于为什么Spark作业会因“打开的文件太多"而失败?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！