如何在不使用临时文件的情况下将流从Web API传递到Azure Blob存储? [英] How to I pass a stream from Web API to Azure Blob storage without temp files?

问题描述

我正在开发一个经常发生文件上载并且可能很大的应用程序.

I am working on an application where file-uploads happen often, and can be pretty big in size.

这些文件将被上载到Web API，然后它将从请求中获取流，并将其传递到我的存储服务，然后将其上载到Azure Blob存储.

Those files are being uploaded to Web API, which will then get the stream from the request, and pass it on to my storage service, that then uploads it to Azure Blob Storage.

我需要确保:

• 没有温度.文件写在Web API实例上
• 在将请求流传递到存储服务之前，它没有完全读入内存(以防止内存不足异常).

我看过

I've looked at this article, that describes how to disable input stream buffering, but as many file uploads from many different users happen simultaneously, it's important that it actually does what it says on the tin.

这是我目前在控制器中所拥有的:

This is what I have in my controller at the moment:

if (this.Request.Content.IsMimeMultipartContent())
{
    var provider = new MultipartMemoryStreamProvider();
    await this.Request.Content.ReadAsMultipartAsync(provider);
    var fileContent = provider.Contents.SingleOrDefault();

    if (fileContent == null)
    {
        throw new ArgumentException("No filename.");
    }

    var fileName = fileContent.Headers.ContentDisposition.FileName.Replace("\"", string.Empty);

    // I need to make sure this stream is ready to be processed by 
    // the Azure client lib, but not buffered fully, to prevent OoM.
    var stream = await fileContent.ReadAsStreamAsync();
}

我不知道如何才能可靠地对此进行测试.

I don't know how I can reliably test this.

编辑:我忘了提一下，直接上传到Blob存储(绕过我的API)是行不通的，因为我正在做一些大小检查(例如，该用户可以上传500mb吗?用了他的配额吗?).

EDIT: I forgot to mention that uploading directly to Blob Storage (circumventing my API) won't work, as I am doing some size checking (e.g. can this user upload 500mb? Has this user used his quota?).

推荐答案

在要点.

这是我使用它的方式，以及一个巧妙的"hack"来获取实际文件大小，而无需先将文件复制到内存中.哦，速度快两倍 (显然).

Here's how I am using it, along with a clever "hack" to get the actual file size, without copying the file into memory first. Oh, and it's twice as fast (obviously).

// Create an instance of our provider.
// See https://gist.github.com/JamesRandall/11088079#file-blobstoragemultipartstreamprovider-cs for implementation.
var provider = new BlobStorageMultipartStreamProvider ();

// This is where the uploading is happening, by writing to the Azure stream
// as the file stream from the request is being read, leaving almost no memory footprint.
await this.Request.Content.ReadAsMultipartAsync(provider);

// We want to know the exact size of the file, but this info is not available to us before
// we've uploaded everything - which has just happened.
// We get the stream from the content (and that stream is the same instance we wrote to).
var stream = await provider.Contents.First().ReadAsStreamAsync();

// Problem: If you try to use stream.Length, you'll get an exception, because BlobWriteStream
// does not support it.

// But this is where we get fancy.

// Position == size, because the file has just been written to it, leaving the
// position at the end of the file.
var sizeInBytes = stream.Position;

Voilá，您具有上传文件的大小，而无需将文件复制到Web实例的内存中.

Voilá, you got your uploaded file's size, without having to copy the file into your web instance's memory.

要获得之前的文件长度，这并不是那么容易，并且我不得不求助于一些不太愉快的方法来获得近似值.

As for getting the file length before the file is uploaded, that's not as easy, and I had to resort to some rather non-pleasant methods in order to get just an approximation.

在BlobStorageMultipartStreamProvider中:

var approxSize = parent.Headers.ContentLength.Value - parent.Headers.ToString().Length;

这给了我一个非常接近的文件大小，只有几百个字节(取决于我猜想的HTTP标头).这对我来说已经足够了，因为我的配额强制执行可以接受被削掉的几个字节.

This gives me a pretty close file size, off by a few hundred bytes (depends on the HTTP header I guess). This is good enough for me, as my quota enforcement can accept a few bytes being shaved off.

这只是炫耀，这是由任务管理器中的准确且高级的性能"选项卡报告的内存占用量.

Just for showing off, here's the memory footprint, reported by the insanely accurate and advanced Performance Tab in Task Manager.

这篇关于如何在不使用临时文件的情况下将流从Web API传递到Azure Blob存储?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！