Mongo聚合和MongoError:异常:BufBuilder尝试将()增长到134217728字节，超过了64MB的限制 [英] Mongo aggregation and MongoError: exception: BufBuilder attempted to grow() to 134217728 bytes, past the 64MB limit

问题描述

我正在尝试通过创建数据的大型json文件以供以后使用，来汇总Mongo集合中的数据以为FreeCodeCamp生成一些统计信息.

I'm trying to aggregate data from my Mongo collection to produce some statistics for FreeCodeCamp by making a large json file of the data to use later.

我遇到标题错误.似乎没有很多相关信息，因此此处的其他帖子也没有答案.我正在使用最新版本的MongoDB和驱动程序.

I'm running into the error in the title. There doesn't seem to be a lot of information about this, and the other posts here on SO don't have an answer. I'm using the latest version of MongoDB and drivers.

我怀疑可能有更好的方法来运行此聚合，但是它在我的收藏集的一部分上运行良好.我的完整收藏集约为7GB.

I suspect there is probably a better way to run this aggregation, but it runs fine on a subset of my collection. My full collection is ~7GB.

我正在通过node aggScript.js > ~/Desktop/output.json运行脚本 这是相关代码:

I'm running the script via node aggScript.js > ~/Desktop/output.json Here is the relevant code:

MongoClient.connect(secrets.db, function(err, database) {
  if (err) {
    throw err;
  }

  database.collection('user').aggregate([
    {
      $match: {
        'completedChallenges': {
          $exists: true
        }
      }
    },
    {
      $match: {
        'completedChallenges': {
          $ne: ''
        }
      }
    },
    {
      $match: {
        'completedChallenges': {
          $ne: null
        }
      }
    },
    {
      $group: {
        '_id': 1, 'completedChallenges': {
          $addToSet: '$completedChallenges'
        }
      }
    }
  ], {
    allowDiskUse: true
  }, function(err, results) {
    if (err) { throw err; }
    var aggData = results.map(function(camper) {
      return _.flatten(camper.completedChallenges.map(function(challenges) {
        return challenges.map(function(challenge) {
          return {
            name: challenge.name,
            completedDate: challenge.completedDate,
            solution: challenge.solution
          };
        });
      }), true);
    });
    console.log(JSON.stringify(aggData));
    process.exit(0);
  });
});

推荐答案

聚合返回包含所有结果数据的单个文档，这限制了可以返回最大BSON文档大小的数据量.

Aggregate returns a single document containing all the result data, which limits how much data can be returned to the maximum BSON document size.

假设您确实确实需要所有这些数据，有两种选择:

Assuming that you do actually want all this data, there are two options:

• 使用aggregateCursor代替aggregate.这将返回一个游标，而不是单个文档，然后可以对其进行迭代
• 添加 $out 阶段作为管道的最后阶段.这告诉mongodb将您的聚合数据写入指定的集合.聚合命令本身不返回任何数据，然后您就可以像查询其他集合一样查询该集合.

• Use aggregateCursor instead of aggregate. This returns a cursor rather than a single document, which you can then iterate over
• add a $out stage as the last stage of your pipeline. This tells mongodb to write your aggregation data to the specified collection. The aggregate command itself returns no data and you then query that collection as you would any other.

这篇关于Mongo聚合和MongoError:异常:BufBuilder尝试将()增长到134217728字节，超过了64MB的限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！