将MongoDB结果流式传输到Express响应 [英] Stream MongoDB results to Express response

问题描述

我试图在没有第三方依赖的情况下执行此操作，因为我认为不需要它们.请注意，由于架构师的裁定，我们必须使用MongoDB本机，而不是Mongoose(不要问！).

I am trying to do this without and 3rd party dependencies, as I don't feel they should be needed. Please note, due to architect ruling we have to use MongoDB native, and not Mongoose (don't ask!).

基本上我有一个getAll函数，它将从单个集合中返回所有文档(基于传入的查询).

Basically I have a getAll function, that will return all documents (based on passed in query) from a single collection.

文档的数量很容易达到几千个，因此，我希望在收到文档时将其流式传输出去.

The number of documents, could easily hit multiple thousand, and thus, I want to stream them out as I receive them.

我有以下代码:

db.collection('documents')
    .find(query)
    .stream({
        transform: (result) => {
            return JSON.stringify(new Document(result));
        }
    })
    .pipe(res);

除了可以破坏文档应位于的数组并响应{...}{...}

Which kind of works, except it destroys the array that the documents should sit in, and it responds {...}{...}

必须有一种正确的方法吗?

There has to be a way of doing this right?

推荐答案

您可以做的是在请求数据库之前显式编写数组res.write("[")的开始，在每个json字符串化对象上放置,，在流末尾写数组res.write("]")的末尾，这可以工作. 但这是不可取的！

What you can do is to write explicitly the start of the array res.write("[") before requesting the database, put a ,, on every json stringified object and on the stream end write the end of the array res.write("]") this can work. But it is not advisable!

JSON.stringify是一个非常慢的操作，您应该尝试尽可能少地使用它.

JSON.stringify is a very slow operation, you should try to use it as less as possible.

更好的方法是使用可流式传输的JSON.stringify实现，例如

A better approach will be to go with a streamable JSON.stringify implementation like json-stream-stringify

const JsonStreamStringify = require('json-stream-stringify');
app.get('/api/users', (req, res, next) => {
   const stream = db.collection('documents').find().stream();
   new JsonStreamStringify(stream).pipe(res);
);

请注意在生产中使用管道，当发生错误时，管道不会破坏源流或目标流.建议使用 pump 或

Be aware of using pipe in production, pipe does not destroy the source or destination stream when errors. It is advisable to go for pump or pipeline in production to avoid memory leaks.

这篇关于将MongoDB结果流式传输到Express响应的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！