内存流和大对象堆 [英] Memorystream and Large Object Heap

问题描述

我必须使用 WCF 通过不可靠的连接在计算机之间传输大文件.

I have to transfer large files between computers on via unreliable connections using WCF.

因为我希望能够恢复文件并且我不想被 WCF 限制我的文件大小，所以我将文件分成 1MB 块.这些块"作为流传输.到目前为止，效果很好.

Because I want to be able to resume the file and I don't want to be limited in my filesize by WCF, I am chunking the files into 1MB pieces. These "chunk" are transported as stream. Which works quite nice, so far.

我的步骤是:

1. 打开文件流
2. 从文件中读取块到字节[]并创建内存流
3. 传输块
4. 返回 2. 直到整个文件发送完毕

我的问题是在第 2 步.我假设当我从一个字节数组创建一个内存流时，它最终会出现在 LOH 上并最终导致内存不足异常.我实际上无法创建此错误，也许我的假设是错误的.

My problem is in step 2. I assume that when I create a memory stream from a byte array, it will end up on the LOH and ultimately cause an outofmemory exception. I could not actually create this error, maybe I am wrong in my assumption.

现在，我不想在消息中发送 byte[]，因为 WCF 会告诉我数组大小太大.我可以更改允许的最大数组大小和/或我的块的大小，但我希望有另一种解决方案.

Now, I don't want to send the byte[] in the message, as WCF will tell me the array size is too big. I can change the max allowed array size and/or the size of my chunk, but I hope there is another solution.

我的实际问题:

• 我当前的解决方案是否会在 LOH 上创建对象，这会导致我出现问题吗?
• 有没有更好的方法来解决这个问题?

顺便说一句:在接收端，我简单地从到达的流中读取较小的块并将它们直接写入文件，因此不涉及大字节数组.

Btw.: On the receiving side I simple read smaller chunks from the arriving stream and write them directly into the file, so no large byte arrays involved.

目前的解决方案:

for (int i = resumeChunk; i < chunks; i++)
{
 byte[] buffer = new byte[chunkSize];
 fileStream.Position = i * chunkSize;
 int actualLength = fileStream.Read(buffer, 0, (int)chunkSize);
 Array.Resize(ref buffer, actualLength);
 using (MemoryStream stream = new MemoryStream(buffer)) 
 {
  UploadFile(stream);
 }
}

推荐答案

我希望这没问题.这是我在 StackOverflow 上的第一个回答.

I hope this is okay. It's my first answer on StackOverflow.

是的，如果您的块大小超过 85000 字节，那么数组将在大对象堆上分配.您可能不会很快耗尽内存，因为您正在分配和释放大小相同的连续内存区域，因此当内存填满时，运行时可以将新块放入旧的、回收的内存区域.

Yes absolutely if your chunksize is over 85000 bytes then the array will get allocated on the large object heap. You will probably not run out of memory very quickly as you are allocating and deallocating contiguous areas of memory that are all the same size so when memory fills up the runtime can fit a new chunk into an old, reclaimed memory area.

我会有点担心 Array.Resize 调用，因为它会创建另一个数组(请参阅 http://msdn.microsoft.com/en-us/library/1ffy6686(VS.80).aspx).如果 actualLength==Chunksize 则这是一个不必要的步骤，因为它适用于除最后一个块之外的所有块.所以我至少建议:

I would be a little worried about the Array.Resize call as that will create another array (see http://msdn.microsoft.com/en-us/library/1ffy6686(VS.80).aspx). This is an unecessary step if actualLength==Chunksize as it will be for all but the last chunk. So I would as a minimum suggest:

if (actualLength != chunkSize) Array.Resize(ref buffer, actualLength);

这应该会删除很多分配.如果actualSize 与chunkSize 不同但仍然> 85000，那么新数组也将分配在大对象堆上，这可能导致它碎片化并可能导致明显的内存泄漏.我相信实际耗尽内存仍然需要很长时间，因为泄漏会非常缓慢.

This should remove a lot of allocations. If the actualSize is not the same as the chunkSize but is still > 85000 then the new array will also be allocated on the Large object heap potentially causing it to fragment and possibly causing apparent memory leaks. It would I believe still take a long time to actually run out of memory as the leak would be quite slow.

我认为更好的实现是使用某种缓冲池来提供数组.您可以推出自己的(这太复杂了)，但 WCF 确实为您提供了一个.我已经稍微重写了您的代码以利用这一点:

I think a better implementation would be to use some kind of Buffer Pool to provide the arrays. You could roll your own (it would be too complicated) but WCF does provide one for you. I have rewritten your code slightly to take advatage of that:

BufferManager bm = BufferManager.CreateBufferManager(chunkSize * 10, chunkSize);

for (int i = resumeChunk; i < chunks; i++)
{
    byte[] buffer = bm.TakeBuffer(chunkSize);
    try
    {
        fileStream.Position = i * chunkSize;
        int actualLength = fileStream.Read(buffer, 0, (int)chunkSize);
        if (actualLength == 0) break;
        //Array.Resize(ref buffer, actualLength);
        using (MemoryStream stream = new MemoryStream(buffer))
        {
            UploadFile(stream, actualLength);
        }
    }
    finally
    {
        bm.ReturnBuffer(buffer);
    }
}

这里假设 UploadFile 的实现可以重写为取一个 int 为 no.要写入的字节数.

this assumes that the implementation of UploadFile Can be rewritten to take an int for the no. of bytes to write.

希望能帮到你

乔

这篇关于内存流和大对象堆的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！