在OSX上的Java文件的限制比在bash低 [英] Java file limit on OSX lower than in bash

问题描述

我增加了最大文件限制对我的MacBook Pro，这样Elasticsearch可以与多个文件的工作，但它不工作。

我运行命令'的ulimit -a，它说打开文件为10万。我可以运行一个简单的shell脚本是这样的：

 出口计数器= 0
而（真）;做触摸的/ tmp / foo的$ {}柜台;出口计数器=`$ expr的柜台+ 1`; DONE
 

和我能够创造大量的文件（超过60,000之前，我杀了脚本）。

不过，使用Java code创建中的/ tmp目录目录的空白子目录RandomAccessFiles，我只能让10232文件之前，我得到的错误：java.io.FileNotFoundException（打开的文件太多）。这里是我的Java code：

 进口java.io. *;
进口的java.util。*;公共类max_open_files {
    公共静态无效的主要（字符串参数... args）抛出异常{
        文件TESTDIR =新的文件（/ tmp目录/ tempsubdir）;
        testDir.mkdirs（）;        清单＆LT;文件＆gt;文件=新的LinkedList＆LT;文件＆gt;（）;
        清单＆LT;＆的RandomAccessFile GT;文件句柄=新的LinkedList＆LT;＆的RandomAccessFile GT;（）;        尝试{
            而（真）{
                文件f =新的文件（TESTDIRTMP+ fileHandles.size（））;
                RandomAccessFile的RAF =新RandomAccessFile的（F，RW）;
                files.add（F）;
                fileHandles.add（RAF）;
            }
        }赶上（例外前）{
            的System.out.println（ex.getClass（）++ ex.getMessage（））;
        }        对于（RAF的RandomAccessFile：文件句柄）raf.close（）        对于（F文件：文件）f.delete（）;        的System.out.println（最大打开文件：+ fileHandles.size（））;
    }
}
 

这java的code是类似于Elasticsearch的code，测试上的文件数量的限制（在MAXOPENFILES方法FileSystemUtils类）。所以Elasticsearch有我的Java程序有同样的问题。

在shell脚本为什么可以比Java code（我和Elasticsearch的）那么多的文件？为什么上的文件的数量高系统的限制没有得到由Java code识别？

更新5月13日下午4:50 CDT：我创建的测试程序的C版本，看看是否限为Java的具体，这似乎是如此。下面的C版本可以打开的文件32765，而在Java code被限制在10232的文件。

 的#include＆LT;＆stdio.h中GT;
＃包括LT＆;＆stdlib.h中GT;INT主（INT ARGC，为const char * argv的）{
    字符名称[100];
    FILE * FP;
    INT NDX;    对于（NDX = 0;＆NDX LT; 50000; NDX ++）{
        sprintf的（名字，/tmp/foo%d.txt，NDX）;
        FP = FOPEN（姓名，W）;
        如果（FP == NULL）{
            fprintf中（标准输出，无法创建文件％d个\\ N，NDX）;
            返回1;
        }
        fprintf中（FP，你好％D，NDX）;
    }    返回0;
}
 

解决方案

<一个href=\"https://developer.apple.com/library/mac/#documentation/Java/Reference/Java_VMOptionsRef/Articles/JavaVirtualMachineOptions.html\"相对=nofollow>为Mac Java虚拟机选项参考

  

-XX： - MaxFDLimit

  
  

将VM从文件描述符限制设置为默认最大克制。默认行为是设置限额由OPEN_MAX，这是10240通常指定的值，这是一个进程可以打开的文件的最大数量。它是可能的，但是，为了增加此限制与sysctl的效用用户指定的值。在这种情况下，你可能想通过-XX：-MaxFDLimit从限制的打开的文件数为10240停止Java虚拟机

I've increased the max files limit on my macbook pro so that Elasticsearch can work with more files, but it isn't working.

I run the command 'ulimit -a' and it says "open files" is 100,000. I can run a simple shell script like this:

export counter=0
while (true) ; do touch "/tmp/foo${counter}" ; export counter=`expr $counter + 1` ; done

And I'm able to create lots of files (over 60,000 before I killed the script).

However, using Java code to create RandomAccessFiles in an empty sub-directory of the "/tmp" directory, I can only make 10,232 files before I get the error: java.io.FileNotFoundException (Too many open files). Here's my Java code:

import java.io.*;
import java.util.*;

public class max_open_files {
    public static void main(String ... args) throws Exception {
        File testDir = new File("/tmp/tempsubdir");
        testDir.mkdirs();

        List<File> files = new LinkedList<File>();
        List<RandomAccessFile> fileHandles = new LinkedList<RandomAccessFile>();

        try {
            while (true) {
                File f = new File(testDir, "tmp" + fileHandles.size());
                RandomAccessFile raf = new RandomAccessFile(f, "rw");
                files.add(f);
                fileHandles.add(raf);
            }
        } catch (Exception ex) {
            System.out.println(ex.getClass() + " " + ex.getMessage());
        }

        for (RandomAccessFile raf : fileHandles) raf.close()

        for (File f : files) f.delete();

        System.out.println("max open files: " + fileHandles.size());
    }
}

This java code is similar to the code in Elasticsearch that tests the limit on the number of files (in the FileSystemUtils class in the maxOpenFiles method). So Elasticsearch has the same problem my Java program has.

Why can the shell script make so many more files than the Java code (mine and Elasticsearch's)? Why does the high system limit on the number of files not get recognized by the Java code?

Update May 13 4:50pm CDT: I created a C version of the test program to see if the limit was Java-specific and it appears to be so. The C version below can open 32,765 files while the Java code is limited to 10,232 files.

#include <stdio.h>
#include <stdlib.h>

int main(int argc, const char *argv) {
    char name[100];
    FILE *fp;
    int ndx;

    for (ndx = 0; ndx < 50000; ndx++) {
        sprintf(name, "/tmp/foo%d.txt", ndx);
        fp = fopen(name, "w");
        if (fp == NULL) {
            fprintf(stdout, "Can not create file %d\n", ndx);
            return 1;
        }
        fprintf(fp, "hello %d", ndx);
    }

    return 0;
}

解决方案

Java Virtual Machine Option Reference for Mac

-XX:- MaxFDLimit

Directs the VM to refrain from setting the file descriptor limit to the default maximum. The default behavior is to set the limit to the value specified by OPEN_MAX, which is 10240. Normally, this is the maximum number of files that a process may have open. It is possible, however, to increase this limit to a user-specified value with the sysctl utility. Under such circumstances, you may want to pass -XX:-MaxFDLimit to stop the Java VM from restricting the number of open files to 10240.

这篇关于在OSX上的Java文件的限制比在bash低的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！