未设置TransferEncodingChunked时HttpClient抛出OutOfMemory异常 [英] HttpClient throws OutOfMemory exception when TransferEncodingChunked is not set

问题描述

为了支持带有进度报告的大文件(实际上是非常大的文件，高达几GB)的上传，我们开始将HttpClient与PushStreamContent结合使用，如

In order to support upload of large (actually very large, up to several gigabytes) files with progress report we started using HttpClient with PushStreamContent, as described here. It works straightforward, we copy bytes between two streams, here is a code example:

    private void PushContent(Stream src, Stream dest, int length)
    {
        const int bufferLength = 1024*1024*10;
        var buffer = new byte[bufferLength];
        var pos = 0;
        while (pos < length)
        {
            var bytes = Math.Min(bufferLength, length - pos);
            src.Read(buffer, 0, bytes);
            dest.Write(buffer, 0, bytes);
            pos += bufferLength;
            dest.Flush();
            Console.WriteLine($"Transferred {pos} bytes");
        }
        dest.Close();
    }

但是在开始时，此代码在传输320 MB之后引发了OutOfMemory异常，即使该进程的内存消耗不是很高(约500 MB).解决此问题的方法是设置TransferEncodingChunked标志:

But in the beginning this code raised OutOfMemory exception after transferring 320 MB even though the memory consumption of the process was not very high (ca. 500 MB). What fixed this problem is setting TransferEncodingChunked flag:

request.Headers.TransferEncodingChunked = true;

不仅我们能够使用此标志设置传输大文件，而且内存消耗减少了90％.

Not only we were able to transfer huge files with this flag set, the memory consumption decreased by 90%.

我还没有找到任何需要使用TransferEncodingChunked的文档，这更多是一个尝试和失败的过程，但是在这种情况下似乎至关重要.仍然令我感到困惑的是，为什么根本引发异常-内存消耗不是很高，是什么原因造成的?

I haven't found any documentation requiring use of TransferEncodingChunked, it was more of a trial and failure process but it seems crucial in this scenario. Still I am puzzled why the exception is thrown at all - the memory consumption is not very high, what causes it?

推荐答案

压缩传输编码

分块传输编码是1.1版中的一种数据传输机制 的超文本传输​​协议(HTTP)，其中数据以 系列的块".它在适当的地方使用了Transfer-Encoding HTTP标头 Content-Length标头的版本， 否则将需要该协议. 1 因为Content-Length标头 不使用时，发件人不需要知道其长度 内容开始发送响应到接收方之前. 发件人可以在开始发送动态生成的内容之前 知道该内容的总大小.

Chunked transfer encoding is a data transfer mechanism in version 1.1 of the Hypertext Transfer Protocol (HTTP) in which data is sent in a series of "chunks". It uses the Transfer-Encoding HTTP header in place of the Content-Length header, which the earlier version of the protocol would otherwise require.1 Because the Content-Length header is not used, the sender does not need to know the length of the content before it starts transmitting a response to the receiver. Senders can begin transmitting dynamically-generated content before knowing the total size of that content.

每个块的大小在块本身之前发送，这样 接收方可以告知接收数据的完成时间 块.数据传输以最后一个长度的块终止 零.

The size of each chunk is sent right before the chunk itself so that the receiver can tell when it has finished receiving data for that chunk. The data transfer is terminated by a final chunk of length zero.

如果我们从逻辑上考虑，该文件将以小块发送，这意味着当您完成一个块时，便会将其从内存中释放出来.最后，您将减少内存消耗，因为您要处理多个小块.

If we think logically, the file is send on little chunks, that means when you finish with a chunk you are freeing it from the memory. In the end you have less memory consumption, because you are dealing with multiple small chunks.

这篇关于未设置TransferEncodingChunked时HttpClient抛出OutOfMemory异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！