从文本文件中读取巨大的字符串 [英] Reading huge line of string from text file

问题描述

我有一个大文本文件，但没有任何换行符。它只包含一个长字符串（1个字符串的大字符串，包含所有ASCII字符），但到目前为止，任何工作都可以，因为我可以用Java读取整行到内存中，但我想知道是否有内存泄漏问题，因为文件变得像5GB +那么大，并且程序无法立即将整个文件读入内存，所以在这种情况下读取此类文件的最佳方法是什么？我们可以把这条巨大的线分成2个部分甚至是多个块吗？

I have a large text file but doesn't have any line break. It just contains a long String (1 huge line of String with all ASCII characters), but so far anything works just fine as I can be able to read the whole line into memory in Java, but i am wondering if there could be a memory leak issue as the file becomes so big like 5GB+ and the program can't read the whole file into memory at once, so in that case what will be the best way to read such file ? Can we break the huge line into 2 parts or even multiple chunks ?

以下是我读取文件的方式

Here's how I read the file

   BufferedReader buf = new BufferedReader(new FileReader("input.txt"));
   String line;
   while((line = buf.readLine()) != null){

   }

推荐答案

单个字符串长度只有20亿个字符，每个字符使用2个字节，所以如果你能读取5 GB的行，它将使用10 GB记忆。

A single String can be only 2 billion characters long and will use 2 byte per character, so if you could read a 5 GB line it would use 10 GB of memory.

我建议您阅读块中的文字。

I suggest you read the text in blocks.

Reader reader = new FileReader("input.txt");
try {
    char[] chars = new char[8192];
    for(int len; (len = reader.read(chars)) > 0;) {
        // process chars.
    }
} finally {
    reader.close();
}

无论文件大小如何，这都将使用大约16 KB。

This will use about 16 KB regardless of the size of the file.

这篇关于从文本文件中读取巨大的字符串的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！