重写WebHostBufferPolicySelector非缓冲文件上传 [英] Overriding WebHostBufferPolicySelector for Non-Buffered File Upload

问题描述

在这篇文章中描述的尝试创建非缓冲文件上传我已经延长System.Web.Http.WebHost.WebHostBufferPolicySelector，覆盖功能UseBufferedInputStream（）的 http://www.strathweb.com/2012/09/dealing-with-大文件 - 在ASP净Web的API / 。当一个文件被发送到我的控制器，我可以在跟踪输出按预期的方式重写功能UseBufferedInputStream（）肯定返回FALSE看到的。但是，使用诊断工具，我可以看到越来越多的文件，内存被上传。

沉重的内存使用量会出现在我的自定义MediaTypeFormatter（像这里的FileMediaFormatter要发生的： http://lonetechie.com/ ）。正是在这样的格式，我想逐步写入传入的文件到硬盘，但我也需要解析JSON和做一些其他的操作与内容类型：多重/表单数据上传。所以我使用的方法HttpContent ReadAsMultiPartAsync（），这似乎是内存增长的源泉。我把跟踪输出前/后的等待，看来，虽然任务是阻止内存使用量相当迅速增加。

一旦我发现通过ReadAsMultiPartAsync返回的部分文件内容（），我以文件内容写入到磁盘使用Stream.CopyTo（）。此写入磁盘不如预期，但不幸的是源文件已经在内存中了这一点。

有没有人有什么可能会错误的想法？似乎ReadAsMultiPartAsync（）正在缓冲整个后的数据;如果这是真的，为什么我们需要无功FILESTREAM =等待fileContent.ReadAsStreamAsync（）来获取文件的内容？是否有另一种方式来完成零件的分裂没有阅读到内存？在我MediaTypeFormatter的code看起来是这样的：

  //保存流，所以我们可以争取/稍后阅读
流流=等待content.ReadAsStreamAsync（）;VAR部分=等待content.ReadAsMultipartAsync（）; //＆LT;  - 内存使用量增长迅速如果（！content.IsMimeMultipartContent（））
{
    抛出新的Htt presponseException（的HTTPStatus code.UnsupportedMediaType）;
}//
// parts.Contents，工艺JSON等拉出来的数据
////发现在多部分内容的文件中的数据
VAR fileContent = parts.Contents.FirstOrDefault（
X =＆GT; 。x.Headers.ContentDisposition.DispositionType.ToLower（）修剪（）==表单数据＆放大器;＆安培;
x.Headers.ContentDisposition.Name.ToLower（）修剪（）==\\+ DATA_CONTENT_DISPOSITION_NAME_FILE_CONTENTS +\\）。//将文件写入磁盘
使用（VAR FILESTREAM =等待fileContent.ReadAsStreamAsync（））
{
    使用（的FileStream toDisk = File.OpenWrite（myUploadedFile.bin））
    {
        （（流）FILESTREAM）.CopyTo（toDisk）;
    }
}
 

解决方案

WebHostBufferPolicySelector 仅在底层的请求是无缓冲指定。这是网络API将在引擎盖下做的：

  IHostBufferPolicySelector policySelector = _bufferPolicySelector.Value;
布尔isInputBuffered = policySelector == NULL？真：policySelector.UseBufferedInputStream（httpContextBase）;
    流的InputStream = isInputBuffered
                  ？ requestBase.InputStream
          ：httpContextBase.ApplicationInstance.Request.GetBufferlessInputStream（）;
 

所以，如果您的实现返回false，则请求是无缓冲。

然而， ReadAsMultipartAsync（）加载到一切的MemoryStream - 因为如果你不指定供应商，它默认为MultipartMemoryStreamProvider。

要获得文件的每一个部分进行处理使用自动保存到磁盘<一个href=\"http://msdn.microsoft.com/en-us/library/system.net.http.multipartformdatastreamprovider%28v=vs.108%29.aspx\">MultipartFormDataStreamProvider （如果你处理文件和表单数据）或<一个href=\"http://msdn.microsoft.com/en-us/library/system.net.http.multipartfilestreamprovider%28v=vs.108%29.aspx\">MultipartFileStreamProvider （如果你处理的只是文件）。

有是上<一个示例href=\"http://www.asp.net/web-api/overview/working-with-http/sending-html-form-data,-part-2\">asp.net或<一个href=\"http://www.strathweb.com/2012/08/a-guide-to-asynchronous-file-uploads-in-asp-net-web-api-rtm/\">here.在这些例子都发生在控制器，但没有任何理由，你为什么会不会也就是格式化使用它。

另一种选择，如果你真的想和流播放是实现从一个自定义类inheritng <一个href=\"https://github.com/ASP-NET-MVC/aspnetwebstack/blob/master/src/System.Net.Http.Formatting/MultipartStreamProvider.cs\">MultipartStreamProvider这将解雇你一旦想要的任何处理，因为它抓住了流的一部分。用法将类似于上述供应商 - 你需要将它传递给 ReadAsMultipartAsync（供应商）法

最后 - 如果你想自杀 - 因为底层的请求流无缓冲理论上你可以使用这样的事情在你的控制器或格式化：

 流流= HttpContext.Current.Request.GetBufferlessInputStream（）;
            字节[] B =新的字节[32 * 1024];
            而（（N = stream.Read（B，0，b.length个））大于0）
            {
                //做的东西与流比特
            }
 

不过，当然这是非常，对于缺乏更好的词中，贫民窟。

In an attempt to create a non-buffered file upload I have extended System.Web.Http.WebHost.WebHostBufferPolicySelector, overriding function UseBufferedInputStream() as described in this article: http://www.strathweb.com/2012/09/dealing-with-large-files-in-asp-net-web-api/. When a file is POSTed to my controller, I can see in trace output that the overridden function UseBufferedInputStream() is definitely returning FALSE as expected. However, using diagnostic tools I can see the memory growing as the file is being uploaded.

The heavy memory usage appears to be occurring in my custom MediaTypeFormatter (something like the FileMediaFormatter here: http://lonetechie.com/). It is in this formatter that I would like to incrementally write the incoming file to disk, but I also need to parse json and do some other operations with the Content-Type:multipart/form-data upload. Therefore I'm using HttpContent method ReadAsMultiPartAsync(), which appears to be the source of the memory growth. I have placed trace output before/after the "await", and it appears that while the task is blocking the memory usage is increasing fairly rapidly.

Once I find the file content in the parts returned by ReadAsMultiPartAsync(), I am using Stream.CopyTo() in order to write the file contents to disk. This writes to disk as expected, but unfortunately the source file is already in memory by this point.

Does anyone have any thoughts about what might be going wrong? It seems that ReadAsMultiPartAsync() is buffering the whole post data; if that is true why do we require var fileStream = await fileContent.ReadAsStreamAsync() to get the file contents? Is there another way to accomplish the splitting of the parts without reading them into memory? The code in my MediaTypeFormatter looks something like this:

// save the stream so we can seek/read again later
Stream stream = await content.ReadAsStreamAsync();  

var parts = await content.ReadAsMultipartAsync(); // <- memory usage grows rapidly

if (!content.IsMimeMultipartContent())
{
    throw new HttpResponseException(HttpStatusCode.UnsupportedMediaType);               
}

//
// pull data out of parts.Contents, process json, etc.
//

// find the file data in the multipart contents
var fileContent = parts.Contents.FirstOrDefault(
x => x.Headers.ContentDisposition.DispositionType.ToLower().Trim() == "form-data" && 
x.Headers.ContentDisposition.Name.ToLower().Trim() == "\"" + DATA_CONTENT_DISPOSITION_NAME_FILE_CONTENTS + "\"");

// write the file to disk
using (var fileStream = await fileContent.ReadAsStreamAsync())
{
    using (FileStream toDisk = File.OpenWrite("myUploadedFile.bin"))
    {
        ((Stream)fileStream).CopyTo(toDisk);
    }
}

解决方案

WebHostBufferPolicySelector only specifies if the underlying request is bufferless. This is what Web API will do under the hood:

IHostBufferPolicySelector policySelector = _bufferPolicySelector.Value;
bool isInputBuffered = policySelector == null ? true : policySelector.UseBufferedInputStream(httpContextBase);
    Stream inputStream = isInputBuffered
                  ? requestBase.InputStream
          : httpContextBase.ApplicationInstance.Request.GetBufferlessInputStream();

So if your implementation returns false, then the request is bufferless.

However, ReadAsMultipartAsync() loads everything into MemoryStream - because if you don't specify a provider, it defaults to MultipartMemoryStreamProvider.

To get the files to save automatically to disk as every part is processed use MultipartFormDataStreamProvider (if you deal with files and form data) or MultipartFileStreamProvider (if you deal with just files).

There is an example on asp.net or here. In these examples everything happens in controllers, but there is no reason why you wouldn't use it in i.e. a formatter.

Another option, if you really want to play with streams is to implement a custom class inheritng from MultipartStreamProvider that would fire whatever processing you want as soon as it grabs part of the stream. The usage would be similar to the aforementioned providers - you'd need to pass it to the ReadAsMultipartAsync(provider) method.

Finally - if you are feeling suicidal - since the underlying request stream is bufferless theoretically you could use something like this in your controller or formatter:

            Stream stream = HttpContext.Current.Request.GetBufferlessInputStream();
            byte[] b = new byte[32*1024];
            while ((n = stream.Read(b, 0, b.Length)) > 0)
            {
                //do stuff with stream bit
            }

But of course that's very, for the lack of better word, "ghetto."

这篇关于重写WebHostBufferPolicySelector非缓冲文件上传的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！