将猫鼬流转换为数组 [英] convert mongoose stream to array

问题描述

我曾使用过mongodb，但对于猫鼬ORM来说还很陌生.我试图从集合中获取数据，explain()输出显示50ms.通过猫鼬获取数据的总时间为9秒.这是查询:

I have worked with mongodb but quite new to mongoose ORM. I was trying to fetch data from a collection and the explain() output was showing 50ms. the overall time it was taking to fetch the data via mongoose was 9 seconds. Here is the query:

Node.find({'dataset': datasetRef}, function (err, nodes){
   // handle error and data here
});

然后，我在要查询的字段上应用了索引.现在explain()输出显示为4ms.但是通过猫鼬检索数据的总时间没有改变.然后我搜索了一下，发现使用lean()可以帮助使mongoose中的读取查询性能与本地mongodb非常接近

Then I applied index on the field I was querying on. The explain() output now showed 4ms. But the total time to retrieve data via mongoose did not change. Then i searched a bit and found that using lean() can help bring the performance of read queries in mongoose quite close to native mongodb

所以我将查询更改为:

Node.find({'dataset': datasetRef})
.lean()
.stream({transform: JSON.stringify})
.pipe(res)

这完全解决了性能问题.但是最终结果是像这样的JSON文档流:

This solved the performance issues completely. But the end result is a stream of JSON docs like this:

{var11: val11, var12: val12}{var21: val21, var22: val22} ...

我如何解析它以形成文档数组?还是我根本不应该使用流?我认为，如果我打算在后端形成数组，则没有必要使用流，因为那样我将不得不等待所有文档都读入内存.但是我也认为，在前端解析和创建整个数组可能会很昂贵.

How do I parse this to form an array of docs ? Or should I not be using stream at all ? In my opinion, there is no point using a stream if I am planning to form the array at backend, since I will then have to wait for all the docs to be read into memory. But I also think that parsing and creating the whole array at front end might be costly.

在这种情况下，如何在不堵塞网络的情况下获得最佳性能?

How can I achieve best performance in this case without clogging the network as well ?

更新

我正在尝试使用直通流解决此问题.但是，我还不能在JSON对象之间插入逗号.请参见下面的代码:

I am trying to solve this problem using a through stream. However, I am not able to insert commas in between the JSON objects yet. See the code below:

res.write("[");

var through = require('through');
var tr = through(
  function write(data){
    this.queue(data.replace(/\}\{/g,"},{"));
  }
);

var dbStream = db.node.find({'dataset': dataSetRef})
.lean()
.stream({'transform': JSON.stringify});

dbStream.on("end", function(){
    res.write("]");
});

dbStream
.pipe(tr)
.pipe(res);

有了这个，我能够在开头得到"["，在结尾得到]".但是，仍然无法将样式} {"替换为}，{".不知道我在做什么错

With this, I am able to get the "[" in the beginning and "]" at the end. However, still not able to get patten "}{" replaced with "},{". Not sure what am I doing wrong

更新2

现在可以弄清楚替换无效的原因.看来，由于我已将transform函数指定为JSON.stringify，因此它一次只能读取一个JSON对象，因此不会遇到模式}{，因为它一次不会选择多个JSON元素.

Now figured out why the replace is not working. It appears that since I have specified the transform function as JSON.stringify, it reads one JSON object at a time and hence never encounter the pattern }{ since it never picks multiple JSON elements at a time.

现在，我已经修改了我的代码，并编写了一个自定义转换函数，该函数执行JSON.stringify，然后在末尾添加逗号.我在这里面临的唯一问题是我不知道它何时是流中的最后一个JSON对象.因为在这种情况下，我不想添加逗号.此刻，一旦遇到结尾，我将添加一个空的JSON对象.但是，这似乎并不令人信服.这是代码:

Now I have modified my code, and written a custom transform function which does JSON.stringify and then appends a comma at the end. The only problem I am facing here is that I don't know when it is the last JSON object in the stream. Because I don't wanna append the comma in that case. At the moment, I append an empty JSON object once the end is encountered. But somehow this does not look like a convincing idea. Here is the code:

res.write("[");
function transform(data){
    return JSON.stringify(data) + ",";
}

var dbStream = db.node.find({'dataset': dataSetRef})
.lean()
.stream({'transform': transform});

dbStream.on("end", function(){
    res.write("{}]");
});

dbStream
.pipe(res);

推荐答案

我在这里面临的唯一问题是我不知道它何时是流中的最后一个JSON对象.

The only problem I am facing here is that I don't know when it is the last JSON object in the stream.

但是您确实知道哪个是第一个.知道了，您可以将逗号放在除第一个对象之外的所有对象之前，而不是附加逗号.为此，请在闭包内设置转换函数:

But you do know which one is first. Knowing that, instead of appending the comma, you can prepend it to every object except the first one. In order to do that, set up your transform function inside a closure:

function transformFn(){

    var first = true;

    return function(data) {

        if (first) {

            first = false;
            return JSON.stringify(data);
        }
        return "," + JSON.stringify(data);
    }
}

现在，您只需调用该函数并将其设置为实际转换即可.

Now you can just call that function and set it as your actual transform.

var transform = transformFn();
res.write("[");
var dbStream = db.node.find({'dataset': dataSetRef})
.lean()
.stream({'transform': transform});

dbStream.on("end", function(){
    res.write("]");
});

dbStream
.pipe(res);

这篇关于将猫鼬流转换为数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！