EMR中打开的文件过多 [英] Too many open files in EMR

问题描述

  EMFILE：在org中打开的文件太多
。 apache.hadoop.io.nativeio.NativeIO.open（Native Method）
 at org.apache.hadoop.io.SecureIOUtils.createForWrite（SecureIOUtils.java:161）
 at org.apache.hadoop.mapred .TaskLog.writeToIndexFile（TaskLog.java:296）
位于org.apache.hadoop.mapred.TaskLog.syncLogs（TaskLog.java:369）
位于org.apache.hadoop.mapred.Child $ 4。运行（Child.java:257）$ java.util.AccessController.doPrivileged中的
（本地方法）$ b $ javax.security.auth.Subject.doAs（Subject.java:396）
 at org.apache.hadoop.mapred.Child.main（Child.java:249）
  c> 

每个reducer约10,000个文件正在创建中。有没有一种方法可以设置每个盒子的ulimit。

我尝试使用以下命令作为引导脚本：
ulimit -n 1000000

但是这完全没有帮助。

我也在bootstrap动作中尝试用下面的代码替换ulimit / usr / lib / hadoop / hadoop-daemon.sh：
$ b

 ＃！/ bin / bash 
 set -e  - x 
 sudo sed -i -e/ ^ ulimit /s|.*|ulimit -n 134217728 | /usr/lib/hadoop/hadoop-daemon.sh 
  

但即使如此，当我们登录到主节点我可以看到ulimit -n返回：32768.
我也确认在/usr/lib/hadoop/hadoop-daemon.sh中进行了所需的更改，它具有：ulimit -n 134217728。

我们有任何hadoop配置吗？
或者是否有解决方法？

我的主要目标是根据每条记录的ID将记录拆分为文件，并且有15亿记录现在可以肯定增加。

任何方式编辑这个文件之前，这个守护进程运行在每个奴隶？

因此，我不是直接将每个文件写入s3，而是将它们本地写入并以1024个文件的批次移至s3。这解决了太多打开的文件问题。

也许当写入s3的文件描述符被打开时，像写入本地文件时一样发布/关闭。任何更好的解释是值得欢迎的。

I am getting the following excpetion in my reducers:

EMFILE: Too many open files
    at org.apache.hadoop.io.nativeio.NativeIO.open(Native Method)
    at org.apache.hadoop.io.SecureIOUtils.createForWrite(SecureIOUtils.java:161)
    at org.apache.hadoop.mapred.TaskLog.writeToIndexFile(TaskLog.java:296)
    at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:369)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:257)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

Per reducer around 10,000 files are being created. Is there a way I can set the ulimit of each box.

I tried using the following command as a bootstrap script: ulimit -n 1000000

But this did not help at all.

I also tried the following in bootstrap action to replace the ulimit command in /usr/lib/hadoop/hadoop-daemon.sh:

#!/bin/bash
set -e -x
sudo sed -i -e "/^ulimit /s|.*|ulimit -n 134217728|" /usr/lib/hadoop/hadoop-daemon.sh

But even then when we log into master node I can see that ulimit -n returns : 32768. I also confirmed that there was the desired change made in /usr/lib/hadoop/hadoop-daemon.sh and it had : ulimit -n 134217728.

Do we have any hadoop configurations for this? Or is there a workaround for this?

My main aim is to split out records into files according to the ids of each record, and there are 1.5 billion records right now which can certainly increase.

Any way to edit this file before this daemon is run on each slave?

解决方案

OK, so it seems that the ulimit set by default in Amazon EMR's setup : 32768 is already way too much and if any job needs more than this then one should revisit their logic. Hence, instead of writing every file directly to s3, I wrote them locally and moved to s3 in batches of 1024 files. This solved too many open files issue.

Perhaps when file descriptors were opened up for writing to s3 weren't getting released/closed as it would when written to local files. Any better explanation to this is welcome.

这篇关于EMR中打开的文件过多的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！