避免聚合16MB限制 [英] Avoid Aggregate 16MB Limit
问题描述
我收集了大约1M个文档.每个文档都有internalNumber
属性,我需要在我的node.js代码中获取所有internalNumber
.
I have a collection of about 1M documents. Each document has internalNumber
property and I need to get all internalNumber
s in my node.js code.
以前我在使用
db.docs.distinct("internalNumber")
或
collection.distinct('internalNumber', {}, {},(err, result) => { /* ... */ })
在节点中.
但是随着集合的增长,我开始出现错误:distinct is too big, 16m cap
.
But with the growth of the collection I started to get the error: distinct is too big, 16m cap
.
现在我要使用聚合.它消耗大量内存,速度很慢,但是可以,因为我只需要在脚本启动时执行一次即可.我已经尝试在Robo 3T GUI工具中进行以下操作:
Now I want to use aggregation. It consumes a lot of memory and it is slow, but it is OK since I need to do it only once at the script startup. I've tried following in Robo 3T GUI tool:
db.docs.aggregate([{$group: {_id: '$internalNumber'} }]);
它可以工作,我想通过以下方式在node.js代码中使用它:
It works, and I wanted to use it in node.js code the following way:
collection.aggregate([{$group: {_id: '$internalNumber'} }],
(err, docs) => { /* ... * });
但是在Node中,我得到一个错误:"MongoError: aggregation result exceeds maximum document size (16MB) at Function.MongoError.create"
.
But in Node I get an error: "MongoError: aggregation result exceeds maximum document size (16MB) at Function.MongoError.create"
.
请帮助克服该限制.
推荐答案
问题在于,本机驱动程序与默认情况下shell方法的工作方式不同,因为"shell"实际上返回的是"cursor"对象,其中本机驱动程序明确"需要此选项.
The problem is that the native driver differs from how the shell method is working by default in that the "shell" is actually returning a "cursor" object where the native driver needs this option "explicitly".
没有光标", .aggregate()
将单个BSON文档作为文档数组返回,因此我们将其变成游标以避免这种限制:
Without a "cursor", .aggregate()
returns a single BSON document as an array of documents, so we turn it into a cursor to avoid the limitation:
let cursor = collection.aggregate(
[{ "$group": { "_id": "$internalNumber" } }],
{ "cursor": { "batchSize": 500 } }
);
cursor.toArray((err,docs) => {
// work with resuls
});
然后,您可以使用常规方法,例如
Then you can use regular methods like .toArray()
to make the results a JavaScript array which on the 'client' does not share the same limitations, or other methods for iterating a "cursor".
这篇关于避免聚合16MB限制的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!