通过Web API将大文件(> 1 GB)上传到Azure Blob存储 [英] upload large files (> 1 GB) to azure blob storage through web api

问题描述

我们有一个托管在.zure应用程序服务中的应用程序(.Net核心)，我们正尝试使用UI中的表单数据通过Web API将大文件上传到Azure blob.我们已经更改了请求长度和API请求超时，即使上传200MB文件，我们仍然面临着连接超时错误

we have an application(.Net core) that is hosted in azure app service and we are trying to upload large files to Azure blob through web API using Form data from UI. We have changed request length and API request timeout still we are facing connection time out errors even while uploading 200MB files

下面是我正在使用的示例代码

below is the sample code I am using

[HttpPost]
[Route("upload")]
[Consumes("multipart/form-data")]
[RequestFormLimits(MultipartBodyLengthLimit = 2147483648)]
public async Task<IHttpActionResult> Upload([FromForm] FileRequestObject fileRequestObject)
{
    var url = "upload_url_to_blob_storage";
    var file = fileRequestObject.Files[0];

    var blob = new CloudBlockBlob(new Uri(url));
    blob.Properties.ContentType = file.ContentType;

    await blob.UploadFromStreamAsync(file.InputStream);

    //some other operations based on file upload
    return Ok();
}


public class FileRequestObject
{
    public List<IFormFile> Files { get; set; }
    public string JSON { get; set; }
    public string BlobUrls { get; set; }

}

推荐答案

根据您的代码，您希望将大文件作为blockblob上载到Azure blob存储.请注意，它有一个限制.有关更多详细信息，请参阅文档

According to your code, you want to upload a large file to Azure blob storage as blockblob. Please note that it has a limitation. For more details, please refer to the document

通过放置Blob"创建的块Blob的最大大小为256 MB 版本2016-05-31及更高版本，旧版本为64 MB.如果你的 对于2016-05-31版和更高版本，blob大于256 MB，或者64 MB 对于较旧的版本，您必须将其作为一组块上传

The maximum size for a block blob created via Put Blob is 256 MB for version 2016-05-31 and later, and 64 MB for older versions. If your blob is larger than 256 MB for version 2016-05-31 and later, or 64 MB for older versions, you must upload it as a set of blocks

因此，如果您想将大文件蔚蓝阻塞块，请使用以下步骤:

So If you want to large files to azure block blob, pleae use the following steps:

• 每块可能8 MB.

• 在每个请求中，它包含一个blockid.

• 在此请求中，您需要按顺序将所有blockid放入正文中.

例如:

[HttpPost]
        [Consumes("multipart/form-data")]
        [RequestFormLimits(MultipartBodyLengthLimit = 2147483648)]
        public async Task<ActionResult> PostAsync([FromForm]FileRequestObject fileRequestObject)
        {
            
          

            string storageAccountConnectionString = "DefaultEndpointsProtocol=https;AccountName=blobstorage0516;AccountKey=UVOOBCxQpr5BVueU+scUeVG/61CZbZmj9ymouAR9609WbqJhhma2N+WL/hvaoNs4p4DJobmT0F0KAs0hdtPcqA==;EndpointSuffix=core.windows.net";
            CloudStorageAccount StorageAccount = CloudStorageAccount.Parse(storageAccountConnectionString);
            CloudBlobClient BlobClient = StorageAccount.CreateCloudBlobClient();
            CloudBlobContainer Container = BlobClient.GetContainerReference("test");
            await Container.CreateIfNotExistsAsync();
            CloudBlockBlob blob = Container.GetBlockBlobReference(fileRequestObject.File.FileName);
            HashSet<string> blocklist = new HashSet<string>();
            var file = fileRequestObject.File;
            const int pageSizeInBytes = 10485760;
            long prevLastByte = 0;
            long bytesRemain = file.Length;

            byte[] bytes;

            using (MemoryStream ms = new MemoryStream())
            {
                var fileStream = file.OpenReadStream();
                await fileStream.CopyToAsync(ms);
                bytes = ms.ToArray();
            }

            // Upload each piece
                do
                {
                    long bytesToCopy = Math.Min(bytesRemain, pageSizeInBytes);
                    byte[] bytesToSend = new byte[bytesToCopy];
                    
                    Array.Copy(bytes, prevLastByte, bytesToSend, 0, bytesToCopy);
                    prevLastByte += bytesToCopy;
                    bytesRemain -= bytesToCopy;

                    //create blockId
                    string blockId = Guid.NewGuid().ToString();
                    string base64BlockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId));

                    await blob.PutBlockAsync(
                        base64BlockId,
                        new MemoryStream(bytesToSend, true),
                        null
                        );

                    blocklist.Add(base64BlockId);

                } while (bytesRemain > 0);

            //post blocklist
            await blob.PutBlockListAsync(blocklist);



            return Ok();
            // For more information on protecting this API from Cross Site Request Forgery (CSRF) attacks, see https://go.microsoft.com/fwlink/?LinkID=717803
        }

public class FileRequestObject
    {
        public IFormFile File { get; set; }
    }

有关更多详细信息，请参阅 https://www.red-gate.com/simple-talk/cloud/platform-as-a-service/azure-blob-storage-part-4-uploading-大斑点/

For more details, please refer to https://www.red-gate.com/simple-talk/cloud/platform-as-a-service/azure-blob-storage-part-4-uploading-large-blobs/

这篇关于通过Web API将大文件(> 1 GB)上传到Azure Blob存储的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！