进行查询时猫鼬降低内存使用量的方法 [英] Way to lower memory usage by mongoose when doing query

问题描述

我正在节点后端工作，试图通过 mongoose 优化对 mongodb 的非常繁重的查询.预期的返回大小相当可观，但出于某种原因，当我发出请求时，节点开始消耗大量内存，例如，单个大请求消耗 200 mb+.

I am working on node backend trying to optimize a very heavy query to mongodb via mongoose. The expected return size is considerable, but for some reason when I make the request, node begins consuming huge amounts of memory, like, 200mb+ for a single big request.

考虑到返回的大小在大多数情况下小于 10mb，这似乎不太对.它也拒绝在完成后释放内存，我知道这可能只是 V8 GC 执行其默认行为，但我担心的是单个 find() 请求消耗了大量内存.

Considering the size of the return is less than 10mb in most cases, this doesn't seem right. It also refuses to let go of memory after it has finished, I know this is probably just the V8 GC doing its default behavior, but what concerns me is the huge amount of memory being consumed for a single find() request.

我通过对 find() 调用的测试将其隔离.调用完成后，它会执行一些后处理，然后将数据发送到回调，所有这些都在匿名函数中.我曾尝试使用查询流而不是 model.find()，但它没有显示出真正的改进.

I've isolated it down through testing to the find() call. Once done with the call, it performs some post processing then sends the data to a callback, all in an anonymous function. I have tried using the querystream instead of the model.find(), but it shows no real improvements.

环顾四周没有产生任何响应，所以我会问，是否有已知的方法来减少，控制或优化猫鼬的内存使用量?有谁知道为什么一次调用使用了这么多多余的内存?

Looking around has not yielded any responses, so I will ask, is there a known way to reduce, control, or optimize the memory usage by mongoose? Does anyone know why so much excess memory is being used for a single call?

编辑

根据 Johnny 和 Blakes 的建议，将 Lean() 与流媒体结合使用，并使用暂停和恢复，极大地提高了运行时间和内存使用率.谢谢！

As per Johnny and Blakes suggestions, using mixture of lean() with streaming, and using pause and resume have improved the runtime and memory usage immensely. Thank you!

推荐答案

您可以使用精益 用于 Mongoose 查询的选项，只要您只需要纯 JavaScript 文档而不是完整的 Mongoose 文档实例.这样可以提高性能并减少内存使用量.

You can use to use the lean option for a Mongoose query as long you only need plain JavaScript documents and not full Mongoose doc instances. This results in faster performance and reduced memory usage.

model.find().lean().exec(function(err, docs) {...});

您还可以将 lean() 与流式结果相结合，这将进一步减少内存使用.

You can also combine lean() with streaming the results which should reduce memory usage even further.

var stream = model.find().lean().stream();

这篇关于进行查询时猫鼬降低内存使用量的方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！