在查询时通过猫鼬降低内存使用量的方法 [英] Way to lower memory usage by mongoose when doing query

问题描述

我正在节点后端上尝试通过mongoose优化对mongodb的非常繁琐的查询.预期的返回大小是相当大的，但是由于某种原因，当我发出请求时，节点开始消耗大量的内存，例如单个大请求的内存为200mb +.

I am working on node backend trying to optimize a very heavy query to mongodb via mongoose. The expected return size is considerable, but for some reason when I make the request, node begins consuming huge amounts of memory, like, 200mb+ for a single big request.

在大多数情况下，考虑到返回的大小小于10mb，这似乎是不对的.它也拒绝在完成后释放内存，我知道这可能只是V8 GC的默认行为，但是让我担心的是，单个find()请求占用了大量内存.

Considering the size of the return is less than 10mb in most cases, this doesn't seem right. It also refuses to let go of memory after it has finished, I know this is probably just the V8 GC doing its default behavior, but what concerns me is the huge amount of memory being consumed for a single find() request.

我已通过测试到find()调用将其隔离.调用完成后，它会执行一些后处理，然后将数据发送到回调，所有这些操作都在匿名函数中.我尝试使用querystream代替model.find()，但是它没有显示出任何实际的改进.

I've isolated it down through testing to the find() call. Once done with the call, it performs some post processing then sends the data to a callback, all in an anonymous function. I have tried using the querystream instead of the model.find(), but it shows no real improvements.

环顾四周并没有产生任何响应，所以我想问一下，有没有一种已知的方法来减少，控制或优化猫鼬的内存使用?有谁知道为什么一个电话要使用这么多的多余内存?

Looking around has not yielded any responses, so I will ask, is there a known way to reduce, control, or optimize the memory usage by mongoose? Does anyone know why so much excess memory is being used for a single call?

编辑

根据Johnny和Blakes的建议，将lean()与流混合使用，并使用暂停和恢复功能极大地改善了运行时和内存使用率.谢谢！

As per Johnny and Blakes suggestions, using mixture of lean() with streaming, and using pause and resume have improved the runtime and memory usage immensely. Thank you!

推荐答案

您可以使用 lean 选项，只要您只需要纯JavaScript文档而不需要完整的Mongoose文档实例即可.这样可以提高性能并减少内存使用.

You can use to use the lean option for a Mongoose query as long you only need plain JavaScript documents and not full Mongoose doc instances. This results in faster performance and reduced memory usage.

model.find().lean().exec(function(err, docs) {...});

您还可以将lean()与流式传输结果结合使用，这将进一步减少内存使用量.

You can also combine lean() with streaming the results which should reduce memory usage even further.

var stream = model.find().lean().stream();

这篇关于在查询时通过猫鼬降低内存使用量的方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！