从分块文件合并大文件部分时出现Java OutOfMemoryError [英] Java OutOfMemoryError while merge large file parts from chunked files

问题描述

当用户上传大文件（> 1 GB）（我正在使用flow.js库）时出现问题，它会在临时目录中创建数十万个小块文件（例如每个100KB）但无法合并到单个文件，由于MemoryOutOfException。当文件低于1 GB时，不会发生这种情况。我知道这听起来很乏味，你可能建议我在我的容器中增加XmX - 但我希望除此之外还有另一个角度。

I have a problem when the user upload large files (> 1 GB) (I'm using flow.js library), it creates hundred of thousand small chunked files (e.g 100KB each) inside temporary directory but failed to merge into single file, due to MemoryOutOfException. This is not happened when the file is under 1 GB. I know it sound tedious and you probably suggest me to increase the XmX in my container-but I want to have another angle besides that.

这是我的代码

private void mergeFile(String identifier, int totalFile, String outputFile) throws AppException{
    File[] fileDatas = new File[totalFile]; //we know the size of file here and create specific amount of the array
    byte fileContents[] = null;
    int totalFileSize = 0;
    int filePartUploadSize = 0;
    int tempFileSize = 0;
    //I'm creating array of file and append the length
    for (int i = 0; i < totalFile; i++) {
        fileDatas[i] = new File(identifier + "." + (i + 1)); //indentifier is the name of the file 
        totalFileSize += fileDatas[i].length();
    }

    try {
        fileContents = new byte[totalFileSize];
        InputStream inStream;
        for (int j = 0; j < totalFile; j++) {
            inStream = new BufferedInputStream(new FileInputStream(fileDatas[j]));
            filePartUploadSize = (int) fileDatas[j].length();
            inStream.read(fileContents, tempFileSize, filePartUploadSize);
            tempFileSize += filePartUploadSize;
            inStream.close();
        }
    } catch (FileNotFoundException ex) {
        throw new AppException(AppExceptionCode.FILE_NOT_FOUND);
    } catch (IOException ex) {
        throw new AppException(AppExceptionCode.ERROR_ON_MERGE_FILE);
    } finally {
        write(fileContents, outputFile);
        for (int l = 0; l < totalFile; l++) {
            fileDatas[l].delete();
        }
    }
}

请显示低效这个方法再一次......只有使用这种方法无法合并的大文件，较小的一个（<1 GB）根本没问题....
如果你不建议我增加我很感激堆内存代替向我显示此方法的基本错误...谢谢...

Please show the "inefficient" of this method, once again... only large files that cannot be merge using this method, smaller one ( < 1 GB) no problem at all.... I appreciate if you do not suggest me to increase the heap memory instead show me the fundamental error of this method... thanks...

谢谢

推荐答案

通过声明整个大小的字节数组，不必在内存中分配整个文件大小。通常在内存中构建连接文件是完全没有必要的。

It's unnecessary to allocate the entire file size in memory by declaring a byte array of the entire size. Building the concatenated file in memory in general is totally unnecessary.

只需打开目标文件的输出流，然后对于要组合的每个文件，只需将每个文件作为输入流读取并写入字节输出流，在完成时关闭每一个。然后，当您完成所有操作后，关闭输出文件。缓冲区的总内存使用量将为几千字节。

Just open up an outputstream for your target file, and then for each file that you are combining to make it, just read each one as an input stream and write the bytes to outputstream, closing each one as you finish. Then when you're done with them all, close the output file. Total memory use will be a few thousand bytes for the buffer.

另外，不要在finally块中进行I / O操作（闭包和东西除外）。

Also, don't do I/O operations in finally block (except closing and stuff).

这是一个你可以玩的粗略例子。

Here is a rough example you can play with.

        ArrayList<File> files = new ArrayList<>();// put your files here
        File output = new File("yourfilename");
        BufferedOutputStream boss = null;
        try 
        {
            boss = new BufferedOutputStream(new FileOutputStream(output));
            for (File file : files) 
            {
                BufferedInputStream bis = null;
                try
                {
                    bis = new BufferedInputStream(new FileInputStream(file));
                    boolean done = false;
                    while (!done)
                    {
                        int data = bis.read();
                        boss.write(data);
                        done = data < 0;
                    }
                }
                catch (Exception e)
                {
                    //do error handling stuff, log it maybe? 
                }
                finally
                {
                    try
                    {
                        bis.close();//do this in a try catch just in case
                    }
                    catch (Exception e)
                    {
                        //handle this
                    }
                }               
            }
        } catch (Exception e) 
        {
            //handle this
        }
        finally
        {
            try 
            {
                boss.close();
            } 
            catch (Exception e) {
                //handle this
            }
        }

这篇关于从分块文件合并大文件部分时出现Java OutOfMemoryError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！