节点Redis发布者占用过多内存 [英] Node redis publisher consuming too much memory

问题描述

我使用 node_redis 库在node中编写了一个小型redis发布者.该程序完成发布1M消息后，将继续保持大约 350 MB 的内存.谁能提供任何线索说明该程序为何需要这么多的内存以及如何释放内存?

I wrote a small redis publisher in node by using node_redis library. After the program is done publishing 1M messages, it continues to hold up around 350 MB of memory. Can anyone provide any clues why the program requires so much memory and how the memory can be released?

以下是代码段-

var redis = require("redis"),
    publisher = redis.createClient();
    var i = 0;
    for (;;) {
        publisher.publish("rChat", i);
        i++;
        if (i == 1000000) {
            console.log("stopped sending messages");
            setTimeout(function(){publisher.end();},1000);
            break;
        }
    }
    setTimeout(function() {
            console.log("Keeping console alive");
        }, 1000000);

推荐答案

这里有两个问题.

为什么程序需要这么多的内存?

我认为这是由于缺乏背压造成的.

I think it is due to the lack of back pressure.

您的脚本仅向Redis发送1M发布命令，但不处理对这些命令的任何答复(因此，它们仅由node_redis丢弃).因为它从不等待任何答复，所以脚本将为所有这些命令在内存中累积大量上下文. node_redis需要保持上下文以跟踪命令，并关联Redis命令和答复. Node.js排队命令的速度比系统将这些命令传送到Redis，对其进行处理，构建回复并将回复传送回node.js的速度要快.因此，上下文正在增长，并且它代表了大量的内存.

Your script just sends 1M publish commands to Redis, but it does not process any reply to these commands (which are therefore just discarded by node_redis). Because it never waits for any reply, the script will accumulate a lot of context in memory for all these commands. node_redis needs to keep a context to keep track of the commands, and associate Redis commands and replies. Node.js is faster to enqueue commands, than the system is to convey those commands to Redis, process them, build replies, and convey replies back to node.js. The context is therefore growing, and it represents a lot of memory.

如果要将内存消耗保持在可接受的水平，则需要降低代码速度，以便有机会让node.js处理Redis答复.例如，以下脚本也处理1M个项目，但是将它们作为1000个项目的批次发布，并等待每1000个项目的答复.因此，它只占用很少的内存(上下文最多包含1000个待处理的命令).

If you want to keep the memory consumption to an acceptable level, you need to throttle down your code to give a chance to node.js to process Redis replies. For instance, the following script also processes 1M items, but it publishes them as batches of 1000 items, and waits for the replies every 1000 items. It therefore consumes very little memory (the context contains at most 1000 pending commands).

var redis = require("redis"),
    publisher = redis.createClient();

function loop( callback ) {
   var count = 0;
   for ( i=0 ; i < 1000; ++i ) {
        publisher.publish("rChat", i, function(err,rep) {
        if ( ++count == 1000 )
            callback();
        });
   }
}

function loop_rec( n, callback ) {
    if ( n == 0 ) {
        callback();
        return;
    }
    loop( function() {
        loop_rec( n-1, callback );
    });
}

function main() {
    console.log("Hello");
    loop_rec(1000, function() {
        console.log("stopped sending messages");
        setTimeout(function(){publisher.end();},1000);
        return;
    });
}

publisher.ping(main)

setTimeout(function() {
    console.log("Keeping console alive");
}, 1000000);

可以释放内存吗?

通常，它不能.作为所有C/C ++程序，node.js使用内存分配器.释放内存后，它不会释放到系统，而是释放到内存分配器.通常，内存分配器无法将未使用的内存返还给系统.请注意，这不是泄漏，因为如果程序执行新的分配，则内存将被重用.

Usually, it cannot. As all C/C++ programs, node.js uses a memory allocator. When memory is freed, it is not released to the system, but to the memory allocator. Generally, the memory allocator is not able to give back the unused memory to the system. Please note it is not a leak, because if the program performs a new allocation, the memory will be reused.

编写实际上可以向系统释放内存的C/C ++程序通常涉及设计自定义内存分配器.很少有C/C ++程序可以做到这一点.最重要的是，node.js包括一个带有v8的垃圾收集器，因此它应该对内存释放策略施加其他约束.

Writing a C/C++ program which can actually release memory to the system generally involves designing a custom memory allocator. Few C/C++ programs do it. On top of that, node.js includes a garbage collector with v8, so it should put additional constraints on the memory releasing policy.

这篇关于节点Redis发布者占用过多内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！