EMR 中打开的文件过多 [英] Too many open files in EMR

问题描述

我的减速器出现以下异常:

I am getting the following excpetion in my reducers:

EMFILE: Too many open files
    at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
    at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
    at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:257)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

每个 reducer 正在创建大约 10,000 个文件.有没有办法设置每个盒子的ulimit.

Per reducer around 10,000 files are being created. Is there a way I can set the ulimit of each box.

我尝试使用以下命令作为引导脚本:ulimit -n 1000000

I tried using the following command as a bootstrap script: ulimit -n 1000000

但这根本没有帮助.

我还在引导操作中尝试了以下操作来替换/usr/lib/hadoop/hadoop-daemon.sh 中的 ulimit 命令:

I also tried the following in bootstrap action to replace the ulimit command in /usr/lib/hadoop/hadoop-daemon.sh:

#!/bin/bash
set -e -x
sudo sed -i -e "/^ulimit /s|.*|ulimit -n 134217728|" /usr/lib/hadoop/hadoop-daemon.sh

但即便如此，当我们登录主节点时，我仍可以看到 ulimit -n 返回:32768.我还确认在/usr/lib/hadoop/hadoop-daemon.sh 中进行了所需的更改，并且它具有:ulimit -n 134217728.

But even then when we log into master node I can see that ulimit -n returns : 32768. I also confirmed that there was the desired change made in /usr/lib/hadoop/hadoop-daemon.sh and it had : ulimit -n 134217728.

我们对此有任何 hadoop 配置吗?或者有解决办法吗?

Do we have any hadoop configurations for this? Or is there a workaround for this?

我的主要目的是根据每条记录的id将记录拆分成文件，现在有15亿条记录，肯定会增加.

My main aim is to split out records into files according to the ids of each record, and there are 1.5 billion records right now which can certainly increase.

在每个从属服务器上运行此守护程序之前，有什么方法可以编辑此文件?

Any way to edit this file before this daemon is run on each slave?

推荐答案

好吧，看来亚马逊 EMR 设置中默认设置的 ulimit : 32768 已经太多了，如果有任何工作需要更多，那么应该重新审视他们的逻辑.因此，我没有将每个文件都直接写入 s3，而是将它们写入本地，然后分批 1024 个文件移动到 s3.这解决了打开文件过多的问题.

OK, so it seems that the ulimit set by default in Amazon EMR's setup : 32768 is already way too much and if any job needs more than this then one should revisit their logic. Hence, instead of writing every file directly to s3, I wrote them locally and moved to s3 in batches of 1024 files. This solved too many open files issue.

也许当打开文件描述符以写入 s3 时，并没有像写入本地文件时那样释放/关闭.欢迎对此进行任何更好的解释.

Perhaps when file descriptors were opened up for writing to s3 weren't getting released/closed as it would when written to local files. Any better explanation to this is welcome.

这篇关于EMR 中打开的文件过多的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！