Java的的OutOfMemoryError在阅读了大量的文本文件 [英] Java OutOfMemoryError in reading a large text file

问题描述

我是新来的Java和工作对阅读非常大的文件，需要一些帮助来理解问题并解决它。我们有一些遗留的code这不得不进行优化，使其运行properly.The文件大小可以从10MB变化只有10GB。当文件开始超越800MB大小唯一麻烦的开始。

I'm new to Java and working on reading very large files, need some help to understand the problem and solve it. We have got some legacy code which have to be optimized to make it run properly.The file size can vary from 10mb to 10gb only. only trouble start when file starting beyond 800mb size.

InputStream inFileReader = channelSFtp.get(path); // file reading from ssh.
byte[] localbuffer = new byte[2048];
ByteArrayOutputStream bArrStream = new ByteArrayOutputStream();

int i = 0;
while (-1 != (i = inFileReader.read(buffer))) {
bArrStream.write(localbuffer, 0, i);
}

byte[] data = bArrStream.toByteArray();
inFileReader.close();
bos.close();

我们所得到的误差

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2271)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)

任何帮助将是AP preciated？

Any help would be appreciated?

推荐答案

尝试使用 java.nio.MappedByteBuffer

<一个href=\"http://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html\">http://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html

您可以映射文件的内容到存储，而无需手动复制它。高级的操作系统提供了内存映射和Java API具有利用该功能。

You can map a file's content onto memory without copying it manually. High-level Operating Systems offer memory-mapping and Java has API to utilize the feature.

如果我的理解是正确的，内存映射不加载文件的全部内容到内存（意为加载和部分必要卸载），所以我想一个10GB的文件不会吃了你的记忆。

If my understanding is correct, memory-mapping does not load a file's entire content onto memory (meaning "loaded and unloaded partially as necessary"), so I guess a 10GB file won't eat up your memory.

这篇关于Java的的OutOfMemoryError在阅读了大量的文本文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！