当我读取 500MB FileStream 时出现 OutOfMemoryException [英] OutOfMemoryException when I read 500MB FileStream
问题描述
我正在使用 Filestream 读取大文件 (> 500 MB)，但出现 OutOfMemoryException.
关于它的任何解决方案.
我的代码是:
using (var fs3 = new FileStream(filePath2, FileMode.Open, FileAccess.Read)){字节[] b2 = ReadFully(fs3, 1024);}public static byte[] ReadFully(Stream stream, int initialLength){//如果我们传递了一个无用的初始长度，只需//使用 32K.如果(初始长度＜1){初始长度 = 32768；}字节 [] 缓冲区 = 新字节 [初始长度];int 读 = 0;整数块；while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0){读取 += 块；//如果我们已经到达缓冲区的末尾，请检查是否有//更多信息如果(读取 == 缓冲区.长度){int nextByte = stream.ReadByte();//流结束?如果是这样，我们就完成了如果(nextByte == -1){返回缓冲区；}//不.调整缓冲区大小，放入我们刚刚创建的字节//阅读并继续byte[] newBuffer = new byte[buffer.Length * 2];Array.Copy(buffer, newBuffer, buffer.Length);newBuffer[read] = (byte)nextByte;缓冲区 = 新缓冲区；阅读++；}}//缓冲区现在太大了.收缩它.字节[] ret = 新字节[读取];Array.Copy(buffer, ret, read);返回 ret;}您展示的代码将 500mb 文件的所有内容读入内存中的一个连续区域.出现内存不足的情况并不奇怪.
解决方案是不要那样做".
你真正想要做什么?
<小时>如果你想完整地读取一个文件，它比你使用的 ReadFully 方法简单得多.试试这个:
using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read)){字节 [] 缓冲区 = 新字节 [fs.Length];int bytesRead = fs.Read(buffer, 0, buffer.Length);//缓冲区现在包含文件的全部内容}但是...使用此代码并不能解决您的问题.它可能适用于 500mb 的文件.它不适用于 750mb 文件或 1gb 文件.在某些时候，您将达到系统上的内存限制，并且您将遇到与开始时相同的内存不足错误.
问题是您试图一次将文件的全部内容保存在内存中.这通常是不必要的，并且随着文件大小的增长注定会失败.当文件大小为 16k 时没有问题.在 500mb 时，这是错误的方法.
这就是为什么我问了好几次，你真正想要做什么?
<小时>听起来您想将文件内容发送到 ASPNET 响应流.这是问题.不是如何将 500mb 的文件读入内存?"但是如何将大文件发送到 ASPNET 响应流?"
对于这个，再一次，它相当简单.
//将文件内容发送到 ASPNET 响应流中使用 (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read)){Response.BufferOutput=假；//防止缓冲字节[]缓冲区=新字节[1024]；int bytesRead = 0;while ((bytesRead = fs.Read(buffer, 0, buffer.Length)) > 0){Response.OutputStream.Write(buffer, 0, bytesRead);}}它所做的是迭代地从文件中读取一个块，然后将该块写入响应流，直到文件中没有更多内容可读取为止.这就是流式IO"的含义.数据通过您的逻辑，但永远不会全部保存在一个地方，就像水流通过水闸一样.在这个例子中，一次在内存中的文件数据永远不会超过 1k(好吧，无论如何，不是由您的应用程序代码保存的.堆栈中还有其他 IO 缓冲区.)>
这是流式 IO 中的常见模式.学习它，使用它.
将数据输出到 ASPNET 的 Response.OutputStream 的一个技巧是设置 BufferOutput = false.默认情况下，ASPNET 会尝试缓冲其输出.在这种情况下(500mb 文件)，缓冲是一个坏主意.将 BufferOutput 属性设置为 false 将阻止 ASPNET 在发送第一个字节之前尝试缓冲所有文件数据.当您知道要发送的文件非常大时，请使用它.数据仍会正确发送到浏览器.
即使这也不是完整的解决方案.您需要设置响应标头等.不过，我想你知道这一点.
I'm using Filestream for read big file (> 500 MB) and I get the OutOfMemoryException.
Any solutions about it.
My Code is:
using (var fs3 = new FileStream(filePath2, FileMode.Open, FileAccess.Read))
{
byte[] b2 = ReadFully(fs3, 1024);
}
public static byte[] ReadFully(Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[initialLength];
int read = 0;
int chunk;
while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
The code you show, reads all content of the 500mb file into a contiguous region in memory. It's not surprising that you get an out-of-memory condition.
The solution is, "don't do that."
What are you really trying to do?
If you want to read a file completely, it's much simpler than the ReadFully method you use. Try this:
using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read))
{
byte[] buffer = new byte[fs.Length];
int bytesRead = fs.Read(buffer, 0, buffer.Length);
// buffer now contains the entire contents of the file
}
But... using this code won't solve your problem. It might work for a 500mb file. It won't work for a 750mb file, or a 1gb file. At some point you will reach the limit of memory on your system and you will have the same out-of-memory error you started with.
The problem is that you are trying to hold the entire contents of the file in memory at one time. This is usually unnecessary, and is doomed to failure as the files grow in size. It's no problem when the filesize is 16k. At 500mb, it's the wrong approach.
This is why I have asked several times, what are you really trying to do ?
Sounds like you want to send the contents of a file out to an ASPNET response stream. This is the question. Not "how to read a 500mb file into memory?" But "how to send a large file to the ASPNET Response stream?"
For this, once again, it's fairly simple.
// emit the contents of a file into the ASPNET Response stream
using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read))
{
Response.BufferOutput= false; // to prevent buffering
byte[] buffer = new byte[1024];
int bytesRead = 0;
while ((bytesRead = fs.Read(buffer, 0, buffer.Length)) > 0)
{
Response.OutputStream.Write(buffer, 0, bytesRead);
}
}
What it does is iteratively read a chunk from the file, and write that chunk to the Response stream, until there is nothing more to read in the file. This is what is meant by "streaming IO". The data passes through your logic, but is never held all in one place, just as a stream of water passes through a sluice. In this example, never is there more than 1k of file data in memory at one time (well, not held by your application code, anyway. There are other IO buffers lower in the stack.)
This is a common pattern in streamed IO. Learn it, use it.
The one trick when pumping data out to ASPNET's Response.OutputStream is to set BufferOutput = false. By default, ASPNET tries to buffer its output. In this case (500mb file), buffering is a bad idea. Setting the BufferOutput property to false will prevent ASPNET from attempting to buffer all the file data before sending the first byte. Use that when you know the file you're sending is very large. The data will still get sent to the browser correctly.
And even this isn't the complete solution. You'll need to set the response headers and so on. I guess you're aware of that, though.
这篇关于当我读取 500MB FileStream 时出现 OutOfMemoryException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
