在 .NET 中解析大型 JSON 文件 [英] Parsing large JSON file in .NET

问题描述

到目前为止，我已经使用了 Json.NET 的JsonConvert.Deserialize(json)"方法，效果很好，老实说，我不需要更多的东西.

I have used the "JsonConvert.Deserialize(json)" method of Json.NET so far which worked quite well and to be honest, I didn't need anything more than this.

我正在开发一个后台(控制台)应用程序，该应用程序不断从不同的 URL 下载 JSON 内容，然后将结果反序列化为 .NET 对象列表.

I am working on a background (console) application which constantly downloads the JSON content from different URLs, then deserializes the result into a list of .NET objects.

 using (WebClient client = new WebClient())
 {
      string json = client.DownloadString(stringUrl);

      var result = JsonConvert.DeserializeObject<List<Contact>>(json);

 }

上面的简单代码片段可能看起来并不完美，但它确实可以完成工作.当文件很大(15,000 个联系人 - 48 MB 文件)时，JsonConvert.DeserializeObject 不是解决方案，并且该行会引发 JsonReaderException 异常类型.

The simple code snippet above doesn't probably seem perfect, but it does the job. When the file is large (15,000 contacts - 48 MB file), JsonConvert.DeserializeObject isn't the solution and the line throws an exception type of JsonReaderException.

下载的 JSON 内容是一个数组，这就是示例的样子.Contact 是反序列化 JSON 对象的容器类.

The downloaded JSON content is an array and this is how a sample looks like. Contact is a container class for the deserialized JSON object.

[
  {
    "firstname": "sometext",
    "lastname": "sometext"
  },
  {
    "firstname": "sometext",
    "lastname": "sometext"
  },
  {
    "firstname": "sometext",
    "lastname": "sometext"
  },
  {
    "firstname": "sometext",
    "lastname": "sometext"
  }
]

我最初的猜测是内存不足.只是出于好奇，我尝试将其解析为 JArray，这也导致了同样的异常.

My initial guess is it runs out of memory. Just out of curiosity, I tried to parse it as JArray which caused the same exception too.

我已经开始深入研究 Json.NET 文档并阅读类似的主题.由于我还没有设法产生一个可行的解决方案，我决定在这里发布一个问题.

I have started to dive into Json.NET documentation and read similar threads. As I haven't managed to produce a working solution yet, I decided to post a question here.

更新:在逐行反序列化时，我得到了同样的错误:[.Path '', line 600003, position 1."所以下载了其中两个并在记事本++中检查了它们.我注意到如果数组长度超过 12,000，则在第 12000 个元素之后，["关闭，另一个数组开始.换句话说，JSON 看起来就像这样:

UPDATE: While deserializing line by line, I got the same error: " [. Path '', line 600003, position 1." So downloaded two of them and checked them in Notepad++. I noticed that if the array length is more than 12,000, after 12000th element, the "[" is closed and another array starts. In other words, the JSON looks exactly like this:

[
  {
    "firstname": "sometext",
    "lastname": "sometext"
  },
  {
    "firstname": "sometext",
    "lastname": "sometext"
  },
  {
    "firstname": "sometext",
    "lastname": "sometext"
  },
  {
    "firstname": "sometext",
    "lastname": "sometext"
  }
]
[
  {
    "firstname": "sometext",
    "lastname": "sometext"
  },
  {
    "firstname": "sometext",
    "lastname": "sometext"
  },
  {
    "firstname": "sometext",
    "lastname": "sometext"
  },
  {
    "firstname": "sometext",
    "lastname": "sometext"
  }
]

推荐答案

正如您在更新中正确诊断的那样，问题是 JSON 有一个结束 ] 紧跟一个开始 [ 开始下一组.这种格式在整体上会使 JSON 无效，这就是 Json.NET 抛出错误的原因.

As you've correctly diagnosed in your update, the issue is that the JSON has a closing ] followed immediately by an opening [ to start the next set. This format makes the JSON invalid when taken as a whole, and that is why Json.NET throws an error.

幸运的是，这个问题似乎经常出现，以至于 Json.NET 实际上有一个特殊的设置来处理它.如果直接使用 JsonTextReader 读取 JSON，可以将 SupportMultipleContent 标志设置为 true，然后使用循环反序列化每个项个人.

Fortunately, this problem seems to come up often enough that Json.NET actually has a special setting to deal with it. If you use a JsonTextReader directly to read the JSON, you can set the SupportMultipleContent flag to true, and then use a loop to deserialize each item individually.

这应该允许您以高效的内存方式成功处理非标准 JSON，而不管有多少数组或每个数组中有多少项.

This should allow you to process the non-standard JSON successfully and in a memory efficient manner, regardless of how many arrays there are or how many items in each array.

    using (WebClient client = new WebClient())
    using (Stream stream = client.OpenRead(stringUrl))
    using (StreamReader streamReader = new StreamReader(stream))
    using (JsonTextReader reader = new JsonTextReader(streamReader))
    {
        reader.SupportMultipleContent = true;

        var serializer = new JsonSerializer();
        while (reader.Read())
        {
            if (reader.TokenType == JsonToken.StartObject)
            {
                Contact c = serializer.Deserialize<Contact>(reader);
                Console.WriteLine(c.FirstName + " " + c.LastName);
            }
        }
    }

完整演示:https://dotnetfiddle.net/2TQa8p

这篇关于在 .NET 中解析大型 JSON 文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！