HttpClient的内存使用量激增，响应较大 [英] HttpClient spike in memory usage with large response

问题描述

我正在使用一个控制台应用程序，该应用程序将视频数据的终结点列表包含在内，发出HTTP请求，并将结果保存到文件中.这些是相对较小的视频.由于我无法控制的问题，其中一个视频非常大(145分钟而不是几秒钟)().

I'm working on a console app that take a list of endpoints to video data, makes an HTTP request, and saves the result to a file. These are relatively small videos. Because of an issue outside of my control, one of the videos is very large (145 minutes instead of a few seconds).

我看到的问题是，在调用该请求后，我的内存使用量猛增至〜1 GB，最终出现任务已取消"错误(大概是因为客户端超时).很好，我不想要这部影片，但是无论我做什么，我所分配的内存仍然很高.我希望能够释放内存.似乎与此有关，任务管理器在此调用之前显示约14 MB的内存使用量，然后此后不断增加.在VS调试器中，我只是看到一个峰值.

The problem I'm seeing is that my memory usage spikes to ~1 GB after that request is called, and I eventually get a "Task was cancelled" error (presumably because the client timed out). This is fine, I don't want this video, but what is concerning is that my allocated memory stays high no matter what I do. I want to be able to release the memory. It seems concerning that Task Manager shows ~14 MB memory usage until this call, then trickles up continuously afterwards. In the VS debugger I just see a spike.

我尝试将所有内容扔到using语句中，在异常情况下重新初始化HttpClient，并在没有运气的情况下手动调用GC.Collect().我正在使用的代码如下所示:

I tried throwing everything in a using statement, re-initializing the HttpClient on exception, manually invoking GC.Collect() with no luck. The code I'm working with looks something like this:

consumer.Received += async (model, ea) =>
{
    InitializeHttpClient(source);
    ...
    foreach(var item in queue)
    {
        await SaveFileFromEndpoint(url, fileName);
        ...
    }
}

和方法:

public void InitializeHttpClient(string source)
{
    ...
    _client = new HttpClient();
    ...
}

public async Task SaveFileFromEndpoint(string endpoint, string fileName)
{
    try
    {
        using (HttpResponseMessage response = await _client.GetAsync(endpoint))
        {
            if (response.IsSuccessStatusCode)
            {
                using(var content = await response.Content.ReadAsStreamAsync())
                using (var fileStream = File.Create($"{fileName}"))
                {
                    await response.Content.CopyToAsync(fileStream);
                }
            }
        }
    }
    catch (Exception ex)
    {

    }
}

这里是我的调试器输出的视图:

Here is a look at my debugger output:

我想我对所看到的内容有一些疑问:

I guess I have a few questions about what I'm seeing:

1. 我看到的内存使用实际上是一个问题吗?
2. 有什么办法可以释放大型HTTP请求分配的内存?
3. 在进行调用和分配内存之前，有什么方法可以查看请求的内容长度吗?到目前为止，在分配实际内存之前，我还无法找到一种方法.

提前感谢您的帮助！

推荐答案

如果使用 HttpCompletionOption.ResponseHeadersRead (与默认的ResponseContentRead相对).这意味着响应流将在下载响应主体之前(而不是在下载之后)交还给您，并且所需的缓冲区要少得多.

If you use HttpClient.SendAsync(HttpRequestMessage, HttpCompletionOption) instead of GetAsync, you can supply HttpCompletionOption.ResponseHeadersRead, (as opposed to the default ResponseContentRead). This means that the response stream will be handed back to you before the response body has downloaded (rather than after it), and will require significantly less buffer to operate.

这篇关于HttpClient的内存使用量激增，响应较大的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！