Json.Net反序列化内存不足问题 [英] Json.Net deserialize out of memory issue

问题描述

我得到了一个Json，其中包含一个数据字段，该数据字段存储一个base64编码的字符串. 此Json已序列化并发送给客户端.

I got a Json, which contains among others a data field which stores a base64 encoded string. This Json is serialized and send to a client.

在客户端，使用newtonsoft json.net解串器来获取Json. 但是，如果数据字段变大(〜400 MB)，则反序列化器将抛出内存不足异常:数组维数超出了支持的范围. 我还在任务管理器中看到，内存消耗确实增长很快.

On client side, the newtonsoft json.net deserializer is used to get back the Json. However, if the data field becomes large (~ 400 MB), the deserializer will throw an out of memory exception: Array Dimensions exceeded supported Range. I also see in Task-Manager, that memory consumption really grows fast.

任何想法为何会这样? json字段或某物有最大大小吗?

Any ideas why this is? Is there a maximum size for json fields or something?

代码示例(简体):

HttpResponseMessage responseTemp = null;
responseTemp = client.PostAsJsonAsync(client.BaseAddress, message).Result;

string jsonContent = responseTemp.Content.ReadAsStringAsync.Result;
result = JsonConvert.DeserializeObject<Result>(jsonContent);

结果类:

public class Result
{

    public string Message { get; set; }
    public byte[] Data { get; set; }

}

更新:

我认为我的问题不是不是串行器，而只是试图处理内存中的这么大的字符串. 当我将字符串读入内存时，应用程序的内存消耗激增.该字符串上的每个操作都相同.目前，我认为我必须找到一种使用流的方法，并停止立即将所有内容读入内存.

I think my problem is not the serializer, but just trying to handle such a huge string in memory. At the point where I read the string into memory, the memory consumption of the application explodes. Every operation on that string does the same. At the moment, I think I have to find a way to work with streams and stop reading the whole stuff into memory at once.

推荐答案

您在这里遇到两个问题:

You have two problems here:

1. 您的JSON响应中有一个单个Base64数据字段，该字段大于约400 MB.

1. You have a single Base64 data field inside your JSON response that is larger than ~400 MB.

您正在将整个响应加载到中间字符串jsonContent中，该字符串甚至更大，因为它嵌入了单个数据字段.

You are loading the entire response into an intermediate string jsonContent that is even larger since it embeds the single data field.

首先，我假设您使用的是64位.如果没有，请开关.

Firstly, I assume you are using 64 bit. If not, switch.

不幸的是，由于Json.NET的 JsonTextReader 无法以与

Unfortunately, the first problem can only be ameliorated and not fixed because Json.NET's JsonTextReader does not have the ability to read a single string value in "chunks" in the same way as XmlReader.ReadValueChunk(). It will always fully materialize each atomic string value. But .Net 4.5 adds the following settings that may help:

1. <gcAllowVeryLargeObjects enabled="true" /> .

此设置最多允许包含一个数组最多只能容纳int.MaxValue个项目.

This setting allows for arrays with up to int.MaxValue entries even if that would cause the underlying memory buffer to be larger than 2 GB. You will still be unable to read a single JSON token of more than 2^31 characters in length, however, since JsonTextReader buffers the full contents of each single token in a private char[] _chars; array, and, in .Net, an array can only hold up to int.MaxValue items.

此设置可以压缩大对象堆，并可以减少由于地址空间碎片而导致的内存不足错误.

This setting allows the large object heap to be compacted and may reduce out-of-memory errors due to address space fragmentation.

但是，第二个问题可以通过流式反序列化来解决，如此问题的答案中所示， Dilip0165 ; John Thiriet的 使用HttpClient和JSON.NET进行高效的api调用 ; 性能提示:优化内存使用 通过Newtonsoft；和 使用新的.NET HttpClient和HttpCompletionOption进行流式传输Tugberk Ugurlu撰写的.ResponseHeadersRead .从这些来源收集信息，您的代码应类似于:

The second problem, however, can be addressed by streaming deserialization, as shown in this answer to this question by Dilip0165; Efficient api calls with HttpClient and JSON.NET by John Thiriet; Performance Tips: Optimize Memory Usage by Newtonsoft; and Streaming with New .NET HttpClient and HttpCompletionOption.ResponseHeadersRead by Tugberk Ugurlu. Pulling together the information from these sources, your code should look something like:

Result result;

var requestJson = JsonConvert.SerializeObject(message); // Here we assume the request JSON is not too large

using (var requestContent = new StringContent(requestJson, Encoding.UTF8, "application/json"))
using (var request = new HttpRequestMessage(HttpMethod.Post, client.BaseAddress) { Content = requestContent })
using (var response = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).Result)
using (var responseStream = response.Content.ReadAsStreamAsync().Result)
{
    using (var textReader = new StreamReader(responseStream))
    using (var jsonReader = new JsonTextReader(textReader))
    {
        result = JsonSerializer.CreateDefault().Deserialize<Result>(jsonReader);
    }
}

或者，使用await:

Result result;

var requestJson = JsonConvert.SerializeObject(message); // Here we assume the request JSON is not too large

using (var requestContent = new StringContent(requestJson, Encoding.UTF8, "application/json"))
using (var request = new HttpRequestMessage(HttpMethod.Post, client.BaseAddress) { Content = requestContent })
using (var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead))
using (var responseStream = await response.Content.ReadAsStreamAsync())
{
    using (var textReader = new StreamReader(responseStream))
    using (var jsonReader = new JsonTextReader(textReader))
    {
        result = JsonSerializer.CreateDefault().Deserialize<Result>(jsonReader);
    }
}

我上面的代码尚未经过全面测试，并且此处和此处. Json.NET的JsonSerializer不支持异步反序列化，使其与HttpClient的异步编程模型有点尴尬.

My code above isn't fully tested, and error and cancellation handling need to be implemented. You may also need to set the timeout as shown here and here. Json.NET's JsonSerializer does not support async deserialization, making it a slightly awkward fit with the asynchronous programming model of HttpClient.

最后，作为使用Json.NET从JSON文件读取巨大的Base64块的一种替代方法，您可以使用此答案. /3744182>通过流处理JSON的某些部分来避免LOH来解析庞大的OData JSON ，以解释如何使用此阅读器和的答案从XmlReader读取流，对它进行base64解码并将结果写入文件 ，以了解如何使用

Finally, as an alternative to using Json.NET to read a huge Base64 chunk from a JSON file, you could use the reader returned by JsonReaderWriterFactory which does support reading Base64 data in manageable chunks. For details, see this answer to Parse huge OData JSON by streaming certain sections of the json to avoid LOH for an explanation of how stream through a huge JSON file using this reader, and this answer to Read stream from XmlReader, base64 decode it and write result to file for how to decode Base64 data in chunks using XmlReader.ReadElementContentAsBase64

这篇关于Json.Net反序列化内存不足问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！