读字符串时OutOfMemoryException异常 [英] OutOfMemoryException when reading a string

问题描述

我有一个C＃.NET系统，需要一个 JSON 数据饲料用它转换为一个对象的 Newtonsoft.Json.JsonConvert.DeserializeObject 转换器。

I have a C# .NET system that takes a JSON data feed and converts it to an object using the Newtonsoft.Json.JsonConvert.DeserializeObject converter.

这过程中完美的作品，只要JSON字符串是具有一定规模（几MB）以下，但只要返回数据是大（几乎100兆），我得到 OutOfMemoryException异常

This process works perfect as long as the JSON string is below a certain size (a few Mb) but as soon as the returned data is large (almost 100Mb) I get the error OutOfMemoryException

此代码的伟大工程为小数据错误：

This code works great for small data:

// WebClient ------------------------------------------------------------------
var _client = new System.Net.WebClient();
var _content = _client.DownloadString(_url);

但在最后一行炸毁（ DownloadString ）

我试图改变这个也适用于小型的数据，但它仍然炸毁了在 ReadToEnd的行当大小的数据增长。

I tried changing to this which also works for small data but it still blew up on the ReadToEnd line when the data grew in size.

using (var _response = (System.Net.HttpWebResponse)_request.GetResponse())
{
    using (System.IO.Stream _dataStream = _response.GetResponseStream())
    {
        using (System.IO.StreamReader _streamReader = new System.IO.StreamReader(_dataStream))
        {
            string _responseFromServer = _streamReader.ReadToEnd();
        }
    }
}

最后，我想这里面工作

Finally I tried this which worked:

StringBuilder _stringBuilder = new StringBuilder();
using (var _response = (System.Net.HttpWebResponse)_request.GetResponse())
{
    using (System.IO.Stream _dataStream = _response.GetResponseStream())
    {
        using (System.IO.StreamReader _streamReader = new System.IO.StreamReader(_dataStream))
        {
            while (!streamReader.EndOfStream)
            {
                char[] _buffer = new char[4096];
                _streamReader.ReadBlock(_buffer, 0, _buffer.Length);
                var _bufferString = new String(_buffer);
                _stringBuilder.Append(_bufferString);
            }
        }
    }
}

不过，它炸毁了与 OutOfMemoryException异常错误，当它到了下一行的位置：

But it blew up with an OutOfMemoryException error when it got to the next line here:

var _results = Newtonsoft.Json.JsonConvert.DeserializeObject<List<MyObject>>(_stringBuilder.ToString());

它不喜欢的ToString（）方法。

这也应声像

string _convertedString = _stringBuilder.ToString();

完整的错误是：

The full error is:

型'的System.OutOfMemoryException的异常出现在
mscorlib.dll中，但用户代码

An exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll but was not handled in user code

本机运行的是64位的窗口16GB内存。

The machine is running 64bit windows with 16Gb of memory.

那么，什么是我的选择？

So, what are my options?

我要的是一个的IQueryable< MyObject的方式> 从（超大型） JSON 字符串

All I want is an IQueryable<MyObject> from a (very large) JSON string.

推荐答案

您的代码基本上是模仿什么的 StreamReader.ReadToEnd 的那样，至少服用4次读大响应（字符串响应本身的存储器中所需要的存储器，所述的StringBuilder的内部缓冲区中，所有的中间临时串的大小和最终的字符串）。

Your code essentially emulates what StreamReader.ReadToEnd does, taking at least 4 times the memory needed to read a large response (the memory of the string response itself, the StringBuilder's internal buffer, the size of all the intermediate temporary strings and the final string).

可以通过从直接与JsonTextReader流反序列可以避免这种情况。从文档样本复制：

You can avoid this by deserializing from the stream directly with a JsonTextReader. Copying from the documentation sample:

using (var json= new JsonTextReader(streamReader))
{
    JsonSerializer serializer = new JsonSerializer();
    return (List<MyObject>)serializer.Deserialize(json, typeof(List<MyObject>));
}

0

O

这篇关于读字符串时OutOfMemoryException异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！