如何在Node.js中处理大量对象 [英] How to process huge array of objects in nodejs

问题描述

我想处理长度为10万左右的数组，而又不会给CPU带来太大的负担.我研究了流，偶然发现了highlandjs，但我无法使其正常工作.

I want to process array of length around 100 000, without putting too much load on CPU. I researched about streams and stumbled upon highlandjs, but i am unable to make it work.

我也尝试过使用Promise和分块处理，但是仍然给CPU带来了很大的负担，如果需要的话程序可能会变慢，但是不应该给CPU带来负担

I also tried using promises and processing in chunks but still it is putting very much load on CPU, program can be slow if needed but should not put load on CPU

推荐答案

使用node.js将Javascript作为单线程运行，如果希望服务器最大程度地响应传入的请求，则需要删除所有占用大量CPU资源的服务器来自主要http服务器进程的代码.这意味着需要以其他方式进行CPU密集型工作.

With node.js which runs your Javascript as single threaded, if you want your server to be maximally responsive to incoming requests, then you need to remove any CPU intensive code from the main http server process. This means doing CPU intensive work in some other process.

有很多不同的方法可以做到这一点:

There are a bunch of different approaches to doing this:

1. 使用child_process模块启动另一个专门用于执行CPU密集型工作的nodejs应用.
2. 对您的应用程序进行聚类，以便您拥有N个不同的进程，这些进程既可以完成CPU密集型工作，又可以处理请求.
3. 创建一个工作队列和许多将处理CPU密集型工作的工作进程.
4. 使用更新的工作者线程将CPU密集型工作移至单独的node.js线程(需要使用节点v12 +才能获得稳定的非实验版本的线程.

1. Use the child_process module to launch another nodejs app that is purposeful built for doing your CPU intensive work.
2. Cluster your app so that you have N different processes that can do both CPU intensive work and handle requests.
3. Create a work queue and a number of worker processes that will handle the CPU intensive work.
4. Use the newer Worker Threads to move CPU intensive work to separate node.js threads (requires node v12+ for stable, non-experimental version of threads).

如果您不经常执行CPU密集型工作，那么#1可能是最简单的.

If you don't do this CPU intensive work very often, then #1 is probably simplest.

如果您出于其他原因(例如处理大量传入请求)而需要扩展，并且您不经常执行CPU密集型工作#2.

If you need scale for other reasons (like handling lots of incoming requests) and you don't do the CPU intensive stuff very often #2.

如果您非常定期地执行CPU密集型工作，并且希望传入请求处理始终具有最高优先级，并且您愿意允许CPU密集型工作花费更长的时间，则可以选择#3(工作队列)或#4(线程)可能是最好的，您可以调整工作人员的数量以优化您的结果.

If you do the CPU intensive stuff pretty regularly and you want incoming request processing to always have the highest priority and you're willing to allow the CPU intensive stuff to take longer, then #3 (work queue) or #4 (threads) is probably the best and you can tune the number of workers to optimize your result.

这篇关于如何在Node.js中处理大量对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！