如何处理内存中非常大的文件(超过8 GB) [英] How can I handle very huge file (more than 8 GB) in memory

问题描述

嗨 我正在尝试合并两个巨大的文件，每个文件的大小约为8 GB.但是当它超过RAM内存时，系统崩溃或引发异常.
该代码的背景是，它使用数组列表从一个巨大的记事本中存储多行，然后在内存中对其进行处理.我也尝试直接在文件中处理它，但是花了几天的时间才能完成它.所以我留下了那个选择.
您能建议我其他任何方法来处理这种大容量内存操作吗?除了使用RAM之外，还有其他方法可以执行这些内存操作，例如通过使用磁盘内存(而不是通过使用任何文件)吗?

我是否应该使用Arraylist以外的其他任何对象，以便可以使用磁盘内存而不是物理内存?

更新问题:
我的问题是一般性的.我如何在没有 RAM约2 GB 的计算机中使用任何文本文件在没有的情况下使用后台文件处理16 GB的文件大小.

我的要求:
它用于附加.
就像这样...
1)对于文件1 中的每一行，选择某些数据
2)完全在 FILE 2 中搜索所选数据，直到数据匹配
如果找到，请选择具有匹配数据的行并进行一些处理，然后将 FILE2 的已处理数据插入 FILE 1 的当前行中的特定位置

我不想直接编辑 FILE1 (因为耗时非常高)，我想从两个文件中读取数据并将其存储在内存中，然后对的每一行进行处理将内存中的FILE 1 与 FILE 2 进行比较，然后将内存中的全部数据写入新的输出文件.

我尝试使用小型文件执行此过程，但运行速度非常快，但由于内存不足而尝试使用大文件时失败了

在此先感谢

Hi I am trying to merge two huge files each of size around 8 GB. But when it exceeds the RAM memory, either the system crashes or it throws an exception.
The background of the code is that it uses an array list to store multiple lines from a very huge notepad and then process the same in the memory. I also tried processing it directly in the file, but it took days to complete it. so I left that option.
Can you please suggest me any other ways to process this type of huge memory operation. Also is there any other ways to do these memory operation apart from using RAM, such as by using disk memory (not by using any File)?

Should I use any other object other than Arraylist so that I can use disk memory rather than physical memory?

Updating the question:
My question in general terms. How can I process a file size of 16 GB in background memory without using any text files to write and read the data in a computer which has a RAM of around 2 GB.

My Requirement:
It is for appending.
It goes like this...
1) For each line in FILE 1, pick certain data
2) Search for the selected data in FILE 2 entirely until the data is matched
If found, select the line which has the matched data and do some process and insert the processed data of FILE2 in the current line of the the FILE 1 at a particular position

Instead of editing FILE1 directly (because time consumption is extreamly very high), I want to read the data from both the file and store it in memory and then do the process for each line of the FILE 1 in memory by comparing it with FILE 2 and then write the entire data in the memory to a new output file.

I tried this process with small sized files and it worked very fast but failed when trying with huge files due to out of memory

Thanks in advance

推荐答案

http://www.daniweb.com/software-development/csharp/threads/279131 [ ^ ]

要读取大文件，可以使用streamReader.
只需在后台线程上执行 reader.ReadToEnd()，并在处理时显示选取框类型的进度条.

或

To read large file you can use streamReader.
Just do reader.ReadToEnd() on a background thread, and display a marquee-type progress bar while processing.

or

//use Buffer with stringbuilder to read textfiles
int bufferSize = 1024;
var sb = new StringBuilder();
var buffer = new Char[bufferSize];
var length = 0L;
var totalRead = 0L;
var count = bufferSize;
using (var sr = new StreamReader(@"C:\Demofile.txt"))
 {
    length = sr.BaseStream.Length;
    while (count > 0)
    {
      count = sr.Read(buffer, 0, bufferSize);
      sb.Append(buffer, 0, count);
      totalRead += count;
    }
 } 

这篇关于如何处理内存中非常大的文件(超过8 GB)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！