使用SharpZipLib压缩大文件，导致内存不足异常 [英] Compress large file using SharpZipLib causing Out Of Memory Exception

问题描述

我有一个453MB的XML文件，我正在尝试使用 SharpZipLib 压缩为ZIP文件.

I have a 453MB XML file which I'm trying to compress to a ZIP using SharpZipLib.

下面是我用来创建zip的代码，但是它导致了OutOfMemoryException.此代码成功压缩了428MB的文件.

Below is the code I'm using to create the zip, but it's causing an OutOfMemoryException. This code successfully compresses a file of 428MB.

不知道为什么会发生异常，因为我不知道为什么，因为我的系统有足够的可用内存.

Any idea why the exception is happening, as I can't see why, as my system has plenty of memory available.

public void CompressFiles(List<string> pathnames, string zipPathname)
{
    try
    {
        using (FileStream stream = new FileStream(zipPathname, FileMode.Create, FileAccess.Write, FileShare.None))
        {
            using (ZipOutputStream stream2 = new ZipOutputStream(stream))
            {
                foreach (string str in pathnames)
                {
                    FileStream stream3 = new FileStream(str, FileMode.Open, FileAccess.Read, FileShare.Read);
                    byte[] buffer = new byte[stream3.Length];
                    try
                    {
                        if (stream3.Read(buffer, 0, buffer.Length) != buffer.Length)
                        {
                            throw new Exception(string.Format("Error reading '{0}'.", str));
                        }
                    }
                    finally
                    {
                        stream3.Close();
                    }
                    ZipEntry entry = new ZipEntry(Path.GetFileName(str));
                    stream2.PutNextEntry(entry);
                    stream2.Write(buffer, 0, buffer.Length);
                }
                stream2.Finish();
            }
        }
    }
    catch (Exception)
    {
        File.Delete(zipPathname);
        throw;
    }
}

推荐答案

您正试图创建一个与文件一样大的缓冲区.而是将缓冲区设置为固定大小，向其中读取一些字节，然后将读取的字节数写入zip文件中.

You're trying to create a buffer as big as the file. Instead, make the buffer a fixed size, read some bytes into it, and write the number of read bytes into the zip file.

这是您的代码，带有4096个字节的缓冲区(以及一些清理):

Here's your code with a buffer of 4096 bytes (and some cleanup):

public static void CompressFiles(List<string> pathnames, string zipPathname)
{
    const int BufferSize = 4096;
    byte[] buffer = new byte[BufferSize];

    try
    {
        using (FileStream stream = new FileStream(zipPathname,
            FileMode.Create, FileAccess.Write, FileShare.None))
        using (ZipOutputStream stream2 = new ZipOutputStream(stream))
        {
            foreach (string str in pathnames)
            {
                using (FileStream stream3 = new FileStream(str,
                    FileMode.Open, FileAccess.Read, FileShare.Read))
                {
                    ZipEntry entry = new ZipEntry(Path.GetFileName(str));
                    stream2.PutNextEntry(entry);

                    int read;
                    while ((read = stream3.Read(buffer, 0, buffer.Length)) > 0)
                    {
                        stream2.Write(buffer, 0, read);
                    }
                }
            }
            stream2.Finish();
        }
    }
    catch (Exception)
    {
        File.Delete(zipPathname);
        throw;
    }
}

特别注意此块:

const int BufferSize = 4096;
byte[] buffer = new byte[BufferSize];
// ...
int read;
while ((read = stream3.Read(buffer, 0, buffer.Length)) > 0)
{
    stream2.Write(buffer, 0, read);
}

这会将字节读入buffer.当没有更多字节时，Read()方法将返回0，这就是我们停止的时间.当Read()成功时，我们可以确定缓冲区中有一些数据，但是我们不知道有多少字节.整个缓冲区可能已满，也可能只有一小部分已满.因此，我们使用读取的字节数read确定要写入ZipOutputStream的字节数.

This reads bytes into buffer. When there are no more bytes, the Read() method returns 0, so that's when we stop. When Read() succeeds, we can be sure there is some data in the buffer but we don't know how many bytes. The whole buffer might be filled, or just a small portion of it. Therefore, we use the number of read bytes read to determine how many bytes to write to the ZipOutputStream.

顺便说一下，该代码块可以由添加到.Net 4.0中的简单语句代替，该语句的功能完全相同:

That block of code, by the way, can be replaced by a simple statement that was added to .Net 4.0, which does exactly the same:

stream3.CopyTo(stream2);

因此，您的代码可能变为:

So, your code could become:

public static void CompressFiles(List<string> pathnames, string zipPathname)
{
    try
    {
        using (FileStream stream = new FileStream(zipPathname,
            FileMode.Create, FileAccess.Write, FileShare.None))
        using (ZipOutputStream stream2 = new ZipOutputStream(stream))
        {
            foreach (string str in pathnames)
            {
                using (FileStream stream3 = new FileStream(str,
                    FileMode.Open, FileAccess.Read, FileShare.Read))
                {
                    ZipEntry entry = new ZipEntry(Path.GetFileName(str));
                    stream2.PutNextEntry(entry);

                    stream3.CopyTo(stream2);
                }
            }
            stream2.Finish();
        }
    }
    catch (Exception)
    {
        File.Delete(zipPathname);
        throw;
    }
}

现在您知道了为什么会出错，以及如何使用缓冲区.

And now you know why you got the error, and how to use buffers.

这篇关于使用SharpZipLib压缩大文件，导致内存不足异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！