在 Node.js/Express 中流式传输/管道 JSON.stringify 输出 [英] Streaming / Piping JSON.stringify output in Node.js / Express

问题描述

我有一个场景，我需要从我的 Node.js/Express RESTful API 返回一个非常大的对象，转换为 JSON 字符串.

I have a scenario where I need to return a very large object, converted to a JSON string, from my Node.js/Express RESTful API.

res.end(JSON.stringify(obj));

但是，这似乎不能很好地扩展.具体来说，它在我连接 1-2 个客户端的测试机上运行良好，但我怀疑此操作可能会杀死 CPU 和许多客户端同时请求大型 JSON 对象时的内存使用情况.

However, this does not appear to scale well. Specifically, it works great on my testing machine with 1-2 clients connecting, but I suspect that this operation may be killing the CPU & memory usage when many clients are requesting large JSON objects simultaneously.

我四处寻找异步 JSON 库，但 我找到的唯一一个似乎有问题(特别是，我收到了 [RangeError]).不仅如此，它还在一大块中返回字符串(例如，对整个字符串调用一次回调，这意味着内存占用不会减少).

I've poked around looking for an async JSON library, but the only one I found seems to have an issue (specifically, I get a [RangeError]). Not only that, but it returns the string in one big chunk (eg, the callback is called once with the entire string, meaning memory footprint is not decreased).

我真正想要的是 JSON.stringify 函数的完全异步管道/流版本，这样它会在数据直接打包到流中时将数据写入......从而节省内存占用，并避免消耗CPU 以同步方式运行.

What I really want is a completely asynchronous piping/streaming version of the JSON.stringify function, such that it writes the data as it is packed directly into the stream... thus saving me both memory footprint, and also from consuming the CPU in a synchronous fashion.

推荐答案

理想情况下，您应该按自己的方式流式传输数据，而不是将所有内容缓冲到一个大对象中.如果你不能改变这一点，那么你需要将 stringify 分解成更小的单元，并允许主事件循环使用 setImmediate.示例代码(我假设主对象有很多顶级属性并使用它们来拆分工作):

Ideally, you should stream your data as you have it and not buffer everything into one large object. If you cant't change this, then you need to break stringify into smaller units and allow main event loop to process other events using setImmediate. Example code (I'll assume main object has lots of top level properties and use them to split work):

function sendObject(obj, stream) {
    var keys = Object.keys(obj);
    function sendSubObj() {
       setImmediate(function(){
          var key = keys.shift();
          stream.write('"' + key + '":' + JSON.stringify(obj[key]));
          if (keys.length > 0) {
            stream.write(',');
            sendSubObj();
          } else {
            stream.write('}');
          }
       });
    })
    stream.write('{');
    sendSubObj();
} 

这篇关于在 Node.js/Express 中流式传输/管道 JSON.stringify 输出的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！