如何将文件动态添加到存储在Azure Blob存储中的zip存档中? [英] How can I dynamically add files to a zip archive stored in Azure blob storage?

问题描述

我在Azure中有一个过程，该过程会生成大量pdf报告文件并将其存储在blob存储中.我不是单独发送所有链接，而是生成一个zip文件并将此链接发送给用户.

I have a process in Azure that generates a large number of pdf report files and stores them in blob storage. Rather than send links to all these individually, I'm generating a zip file and sending this link to users.

此过程全部在一个过程中完成，并且运行良好.最近，在将文件添加到zip存档时遇到了OutOfMemory异常错误，我一直在努力寻找解决方案.

This process is all done in a single process, and has been working fine. Lately, I've been getting OutOfMemory exception errors when adding files to the zip archive and I'm struggling to find a solution.

下面是我用来创建zip文件的代码(注意:使用SharpLibZip库).当前，在添加大约45个文件(每个文件约3.5Mb)后，它会失败并显示OutOfMemoryException(PDF).当我上一行时发生故障:zipStream.PutNextEntry(newEntry).

Below is the code I use to create the zip file (note: using the SharpLibZip library). Currently, it fails with an OutOfMemoryException after adding about 45 files of about 3.5Mb per file (PDF). The failure occurs when I hit the line: zipStream.PutNextEntry(newEntry).

有人知道我可以如何改善这一过程吗?在这个级别上，似乎压缩文件很小.

Does anyone know how I could improve this process? It seems to small a zip file to fail at this level.

Using outputMemStream As New MemoryStream()

    Using zipStream As New ICSharpCode.SharpZipLib.Zip.ZipOutputStream(outputMemStream)
          zipStream.SetLevel(7)

          Dim collD3 As UserSurveyReportCollection = GetFileList(RequestID)

          For Each entityD2 As UserSurveyReport In collD3

              Try
                  Dim strF As String = entityD2.FileLocation

                 'Download blob as memorystream and add this stream to the zip file
                 Dim msR As New MemoryStream 
                 msR = objA.DownloadBlobAsMemoryStream(azureAccount, ReportFolder, entityD2.FileName)
                 msR.Seek(0, SeekOrigin.Begin)

                'Determine file name used in zip file archive for item
                 Dim strZipFileName As String = DetermineZipSourceName(entityD2, strFolder, strFileName)

                 'Add MemoryStream to ZipFile Stream
                 Dim newEntry As ICSharpCode.SharpZipLib.Zip.ZipEntry = New ICSharpCode.SharpZipLib.Zip.ZipEntry(strZipFileName)
                 newEntry.DateTime = DateTime.Now

                 zipStream.PutNextEntry(newEntry)
                 msR.CopyTo(zipStream)
                 zipStream.CloseEntry()

                 msR = Nothing
                 zipStream.Flush()

                 intCounter += 1

        End If

    Catch exZip As Exception

    End Try

  Next


    zipStream.IsStreamOwner = False
    zipStream.Finish()
    zipStream.Close()

    outputMemStream.Position = 0

    Dim bytes As Byte() = outputMemStream.ToArray()
    result.Comment = objA.UploadBlob(bytes, azureAccount, ReportFolder, entityReport.FileName).AbsolutePath


    End Using
  End Using

推荐答案

对于使用C#交易并希望将大型zip文件写入blob存储的任何人:

For anyone who deals in C# and wants to write a large zip file to blob storage:

var blob = container.GetBlockBlobReference(outputFilename);
using (var stream = await blob.OpenWriteAsync())
using (var zip = new ZipArchive(stream, ZipArchiveMode.Create))
{
    for (int i = 0; i < 2000; i++)
    {
        using (var randomStream = CreateRandomStream(2))
        {
            var entry = zip.CreateEntry($"{i}.zip", CompressionLevel.Optimal);
            using (var innerFile = entry.Open())
            {
                await randomStream.CopyToAsync(innerFile);
            }
        }
    }
}

这出奇地好.流到Azure时，应用程序内存约为20Mb，CPU占用率非常低.我已经创建了非常大的输出文件(> 4.5Gb)，没有问题

This works surprisingly well. App memory about 20Mb with very low CPU as it streams to Azure. I've created very large output files (> 4.5Gb) with no problem

这篇关于如何将文件动态添加到存储在Azure Blob存储中的zip存档中?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！