很长的对象数组的NodeJS JSON.stringify错误“无效的字符串长度"； [英] NodeJS JSON.stringify of very long array of objects errors "invalid string length"

问题描述

我有很长的对象数组，大约有100,000个项目，在将其写入文件之前，我将数据传递到JSON.stringify

我收到此错误:

JSON.stringify(
     ^
RangeError: Invalid string length

如何成功地对非常大的json对象进行字符串化处理?

更新:

自从我最初写这个问题以来，我对处理大量数据变得更加熟悉，并且我更喜欢在此过程中使用Apache Spark.在Spark中，习惯上不使用大型JSON数组，而是使用包含JSON长字符串(每行一个对象)的流或文件.文件将如下所示:

{"name":"kristian", "favoriteSong":"enter sandman"}
{"name":"joshua", "favoriteSong":"penny lane"}
{"name":"susan", "favoriteSong":"all about that bass"}

这很好，因为:

• 您可以任意处理TB级的文件，并且只要逻辑上只是以某个换行号N来破坏文件，就可以像轻松地进行一些无逻辑的文件拆分并将其变成一千个较小的文件一样. li>
• 这样最好，您永远不会遇到由未关闭的JSON数组引起的数据文件损坏的潜在可怕问题.
• 文件中的大型JSON数组往往没有换行符，因此大型文件很难在VIM或其他编辑器中打开，因为它试图将整个内容加载到内存中，而不是带有换行符的文件，可以轻松地将前10行左右的内容加载到屏幕上，而不会出现问题.

解决方案

如何成功地对非常大的json对象进行字符串化处理?

部分?例如，将数组分成较小的部分，在较小的部分上使用JSON.stringify，然后将每个段附加到文件中(您必须执行一些[，,和]处理). /p>

I've got a very long array of objects that is about 100,000 items in size, and just before I got to write it to file, I pass the data into JSON.stringify

I get this error:

JSON.stringify(
     ^
RangeError: Invalid string length

How can I stringify a very large json object successfully?

Update:

since I initially had written this question, I've become more familiar with processing large amounts of data, and I prefer to use Apache Spark for this process. In Spark, it is customary to not use large JSON arrays, but rather a stream or file containing long strings of JSON, one object per line. A file would look like this:

{"name":"kristian", "favoriteSong":"enter sandman"}
{"name":"joshua", "favoriteSong":"penny lane"}
{"name":"susan", "favoriteSong":"all about that bass"}

This is good because:

• you can arbitrarily deal with files that are terabytes large, and just as easily do some logicless file splitting and turn that into a thousand smaller files, were the logic is to simply break the file at some line break number N.
• This is better you wont ever have the potentially horrible issue of having a corrupted data file on your hands, caused by an un-closed JSON array.
• a large JSON array in a file tends to not have line breaks, and thus large files are a pain to open in VIM or other editors because it tries to load the whole thing into memory, as opposed to a file with linebreaks, where it can easily load the first 10 lines or so to the screen without issue.

解决方案

How can I stringify a very large json object successfully?

In parts? E.g., break the array up into smaller parts, use JSON.stringify on the smaller parts, and append each segment to the file (you'll have to do some [, ,, and ] handling).

这篇关于很长的对象数组的NodeJS JSON.stringify错误“无效的字符串长度"；的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！