填充 MemoryStream 时出现 OutOfMemoryException:16GB 系统上的 256MB 分配 [英] OutOfMemoryException while populating MemoryStream: 256MB allocation on 16GB system

问题描述

我在安装了 16GB RAM 的 64 位 Windows 7 机器上的开发 IIS 服务器(来自 VS2010 IDE)上运行以下方法:

I'm running the following method on my development IIS server (from VS2010 IDE) on a 64-bit Windows 7 machine with 16GB of installed RAM:

public static MemoryStream copyStreamIntoMemoryStream(Stream stream)
{
    long uiLen = stream.Length;
    byte[] buff = new byte[0x8000];

    int nSz;
    MemoryStream ms = new MemoryStream();
    try
    {
        while ((nSz = stream.Read(buff, 0, buff.Length)) != 0)
        {
            ms.Write(buff, 0, nSz);
        }
    }
    finally
    {
        Debug.WriteLine("Alloc size=" + ms.Length);
    }

    return ms;
}

我在这一行得到 System.OutOfMemoryException:

ms.Write(buff, 0, nSz);

当分配了 268435456 个字节时抛出:

That is thrown when 268435456 bytes are allocated:

分配大小=268435456

Alloc size=268435456

这是 0x10000000 或 256 MB.所以我想知道是否需要设置一些全局设置才能使其工作?

which is 0x10000000 or 256 MB. So I'm wondering if there's some global setting that I need to set to make it work?

这是项目的配置设置的屏幕截图:

Here's a screenshot of the configuration setting for the project:

推荐答案

简短回答 - 开发服务器是 32 位进程.

Short answer - dev server is 32bit process.

为什么只有 256Mb?"的长答案

Long answer for "why just 256Mb?"

首先，让我们了解它是如何工作的.

First of all, let's understand how it works.

MemoryStream 有内部 byte[] 缓冲区来保存所有数据.它无法预测此缓冲区的确切大小，因此它只是使用某个初始值对其进行初始化.

MemoryStream has internal byte[] buffer to keep all the data. It cannot predict exact size of this buffer, so it just initializes it with some initial value.

Position 和 Length 属性不反映实际缓冲区大小 - 它们是反映写入字节数的逻辑值，并且很容易小于实际物理缓冲区大小.

Position and Length properties don't reflect actual buffer size - they are logical values to reflect how many bytes is written, and easily may be smaller than actual physical buffer size.

当这个内部缓冲区不能容纳所有数据时，它应该重新调整大小"，但在现实生活中这意味着创建新缓冲区 两倍于以前的大小，然后复制数据从旧缓冲区到新缓冲区.

When this internal buffer can not fit all the data, it should be "re-sized", but in real life it means creating new buffer twice as size as previous one, and then copying data from old buffer to new buffer.

因此，如果您的缓冲区的长度为 256Mb，并且您需要写入新数据，这意味着 .Net 需要找到另一个 512Mb 的数据块 - 将所有其他数据块都准备好，因此堆应该在收到 OutOfMemory 时分配内存时至少 768Mb.

So, if the length of your buffer is 256Mb, and you need new data to be written, this means that .Net need to find yet another 512Mb block of data - having all the rest in place, so heap should be at least 768Mb on the moment of memory allocation when you receive OutOfMemory.

另请注意，默认情况下，.Net 中的单个对象(包括数组)的大小不会超过 2Gb.

Also please notice that by default no single object, including arrays, in .Net can take more than 2Gb in size.

好的，这是模拟正在发生的事情的示例:

Ok, so here is the sample piece which simulates what's happening:

        byte[] buf = new byte[32768 - 10];

        for (; ; )
        {
            long newSize = (long)buf.Length * 2;
            Console.WriteLine(newSize);

            if (newSize > int.MaxValue)
            {
                Console.WriteLine("Now we reach the max 2Gb per single object, stopping");
                break;
            }

            var newbuf = new byte[newSize];
            Array.Copy(buf, newbuf, buf.Length);
            buf = newbuf;
        }

如果它内置 x64/AnyCPU 并从控制台运行 - 一切正常.

If it built in x64/AnyCPU and runs from console - everything is ok.

如果它跨 x86 构建 - 它在控制台中失败.

If it built across x86 - it fails in console.

如果你说 Page_Load，内置 x64，并从 VS.Net Web 服务器打开 - 它会失败.

If you put it to say Page_Load, built in x64, and open from VS.Net web server - it fails.

如果您对 IIS 执行相同操作 - 一切正常.

If you do the same with IIS - everything is ok.

希望这会有所帮助.

这篇关于填充 MemoryStream 时出现 OutOfMemoryException:16GB 系统上的 256MB 分配的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！