Node.js分叉的子返回消息限制及其解决方法 [英] Node.js forked child return message limit, and ways around it

问题描述

我在节点上做了一些繁重的统计工作，所以我将计算分叉给了一个子进程，该子进程将处理所有mongo查询并遍历数据等，最后将结果发送回母进程.相对较小的数据集，一切正常.

I am doing some heavy statistic in node, so I forked the computation off to a child process where the child process will handle all the mongo query and loop through data etc and finally sends the result back to mother process. It's working all fine with relatively small data set.

但是，当数据足够大时，通信似乎被阻塞，或者在发回过程中出了点问题.返回数据是相当大的HighChart对象，其中包含超过200k数据点，我能够在子进程调用process.send(data)之前记录消息，并且子进程上的cpu活动几乎降至0，母进程的CPU使用率提高了很多，只停留在那儿而不下降.到那时，该过程似乎只是挂起(不是挂起，因为它仍在响应其他请求)，但是该过程是on.'(message'，function(){});即使我只是让它坐在那里运行也不会触发.

But when the data gets large enough, the communication seems to be choked or something went wrong during the send back. The return data is fairly large HighChart Object which contains over 200k data point, I am able to log the message right before the child process invoke process.send(data), and the cpu activity just drop to almost 0 on the child process, which the mother process cpu usage got cranked way high, and just stay there and not dropping down. At that point the process seems just hang ( well not hang, cuz it's still responding to other request ), but the process.on('message',function(){}); is not triggered even I just let it sit there and run.

IPC可以处理节点的大小是否受到限制?无论如何，还是有一些破解?有人建议使用TCP或其他IP协议代替IPC，但这有点使管理儿童寿命问题变得复杂.

Is there a limit on the size where the IPC can handle for node? Is there anyway around this or some hack to it? Some people suggested to use TCP or other IP protocol instead of IPC, but it kinda complicates the issue with managing the child life span a bit.

除了使用console.log之外，还有没有免费的工具可以使调试节点更加容易?

Also is there any free tool that makes debugging node a bit easier beside just using console.log ?

非常感谢您的提前帮助

关于.G

推荐答案

伙计，一些人登陆此页面，并认为它很有用.

Folks, a few people are landing on this page and thinking that it is useful.

process.send()不使用System V IPC，因此Penryn链接到System V消息大小的上述信息完全无关.

process.send() does not use System V IPC, so the information above by Penryn linking to System V message sizes is completely irrelevant.

process.send()消息的大小没有基本限制，除了内存和cpu可用性以创建和json编码/解码此类消息外.与node/javascript一样，JSON.parse()是同步的，因此在极大的JSON数据缓冲区上调用它会阻塞node.如果存在问题，请跨管道流JSON，并使用流JSON解码库之一-总体性能会较差，但总体响应性会更好，因为您不会长时间阻塞解析.

There is no fundamental limit to process.send() message sizes, other than memory and cpu availability to create and json encode/decode such a message. As usual with node/javascript, JSON.parse() is synchronous, so calling it on extremely large buffers of JSON data will block node. If this is a problem, stream the JSON across a pipe, and use one of the streaming JSON decoding libraries - the overall performance will be poorer, but the overall responsiveness will be better because you won't get long blocking parses.

如果有人可以找到所谓的消息大小限制"的可复制示例，请通过github.com/nodejs/node将其作为错误报告给我们.

If anybody can find a reproduceable example of a so-called "message size limit", please report it as a bug to us at github.com/nodejs/node

我所见过的所有实例都是用户的应用程序代码中的错误.

All instances I've seen of this have been errors in the user's application code.

这篇关于Node.js分叉的子返回消息限制及其解决方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！