base64转换后如何释放内存 [英] How to free up memory after base64 convert

问题描述

我正在尝试流式传输文件的内容. 该代码适用于较小的文件，但是对于较大的文件，出现内存不足"错误.

I am trying to stream the contents of a file. The code works for smaller files, but with larger files, I get an Out of Memory error.

public void StreamEncode(FileStream inputStream, TextWriter tw)
{
    byte[] base64Block = new byte[BLOCK_SIZE];
    int bytesRead = 0;

    try
    {
        do
        {
            // read one block from the input stream
            bytesRead = inputStream.Read(base64Block, 0, base64Block.Length);

            // encode the base64 string
            string base64String = Convert.ToBase64String(base64Block, 0, bytesRead);

            // write the string
            tw.Write(base64String);

        } while (bytesRead == base64Block.Length);
    }
    catch (OutOfMemoryException)
    {
        MessageBox.Show("Error -- Memory used: " + GC.GetTotalMemory(false) + " bytes");
    }
}

我可以找出问题所在，并观察所使用的内存在循环时的增长情况.
问题似乎出在给Convert.ToBase64String()的调用上.

I can isolate the problem and watch the memory used grow as it loops.
The problem seems to be the call to Convert.ToBase64String().

如何释放转换后的字符串的内存?

How can I free the memory for the converted string?

从这里向下编辑...这是更新. 我还为此创建了一个新的线程 –对不起，我猜想那不是正确的事情.

Edited from here down ... Here is an update. I also created a new thread about this -- sorry I guess that was not the right thing to do.

感谢您的出色建议.根据建议，我缩小了用于读取文件的缓冲区大小，看起来内存消耗更好，但是我仍然看到OOM问题，并且看到文件大小小至5MB的问题.我可能想处理大十倍的文件.

Thanks for your great suggestions. From the suggestions, I shrunk the buffer size used to read from the file, and it looks like memory consumption is better, but I'm still seeing an OOM problem, and I'm seeing this problem with files sizes as small as 5MB. I potentially want to deal with files ten times larger.

我的问题现在似乎与TextWriter的使用有关.

My problem seems now to be with the use of TextWriter.

我按如下方式创建一个请求[进行一些编辑以缩小代码范围]:

I create a request as follows [with a few edits to shrink the code]:

HttpWebRequest oRequest = (HttpWebRequest)WebRequest.Create(new Uri(strURL));
oRequest.Method = httpMethod;
oRequest.ContentType = "application/atom+xml";
oRequest.Headers["Authorization"] = getAuthHeader();
oRequest.ContentLength = strHead.Length + strTail.Length + longContentSize;
oRequest.SendChunked = true;

using (TextWriter tw = new StreamWriter(oRequest.GetRequestStream()))
{
    tw.Write(strHead);
    using (FileStream fileStream = new FileStream(strPath, FileMode.Open, 
           FileAccess.Read, System.IO.FileShare.ReadWrite))
    {
        StreamEncode(fileStream, tw);
    }
    tw.Write(strTail);
}
.....

哪个要调用例程:

public void StreamEncode(FileStream inputStream, TextWriter tw)
{
    // For Base64 there are 4 bytes output for every 3 bytes of input
    byte[] base64Block = new byte[9000];
    int bytesRead = 0;
    string base64String = null;

    do
    {
        // read one block from the input stream
        bytesRead = inputStream.Read(base64Block, 0, base64Block.Length);

        // encode the base64 string
        base64String = Convert.ToBase64String(base64Block, 0, bytesRead);

        // write the string
        tw.Write(base64String);


    } while (bytesRead !=0 );

}

由于潜在的大量内容，我是否应该使用TextWriter以外的其他东西?能够创建请求的整个有效负载似乎非常方便.

Should I use something other than TextWriter because of the potential large content? It seems very convenient for being able to create the whole payload of the request.

这是完全错误的方法吗?我希望能够支持非常大的文件.

Is this totally the wrong approach? I want to be able to support very large files.

推荐答案

如果使用32 kB或更大的BLOCK_SIZE，则将创建85 kB或更大的字符串，这些字符串分配在大对象上堆.寿命短的对象应该驻留在常规堆中，而不是大对象堆中，这可能是内存问题的原因.

If you use a BLOCK_SIZE that is 32 kB or more, you will be creating strings that are 85 kB or more, which are allocated on the large objects heap. Short lived objects should live in the regular heaps, not the large objects heap, so that may be the reason for the memory problems.

此外，我看到了代码的两个潜在问题:

Also, I see two potential problems with the code:

• base64编码在字符串末尾使用填充，因此，如果将流切成位并转换为base64字符串，然后将字符串写入流，则不会以a结尾单个base64流.

• The base64 encoding uses padding at the end of the string, so if you chop up a stream into bits and convert to base64 strings, and then write the strings to a stream, you don't end up with a single base64 stream.

检查使用Read方法读取的字节数是否与请求的字节数相同，这不是检查流末尾的正确方法. Read方法可能会在感觉不到的任何时候读取少于请求的字节，并且检查流的结尾的正确方法是该方法返回零时.

Checking if the number of bytes read using the Read method is the same as the number of requested bytes is not the proper way of checking for the end of the stream. The Read method may read less bytes than requested any time it feels like it, and the correct way to check for the end of the stream is when the method returns zero.

这篇关于base64转换后如何释放内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！