如何在nodejs中处理大量对象 [英] How to process huge array of objects in nodejs

问题描述

我想处理长度约为 100 000 的数组，而不会给 CPU 带来太多负载.我研究了流并偶然发现了 highlandjs，但我无法使其工作.

I want to process array of length around 100 000, without putting too much load on CPU. I researched about streams and stumbled upon highlandjs, but i am unable to make it work.

我也尝试过使用 Promise 和分块处理，但它仍然给 CPU 带来了很大的负载，如果需要，程序可能会很慢，但不应该给 CPU 带来负载

I also tried using promises and processing in chunks but still it is putting very much load on CPU, program can be slow if needed but should not put load on CPU

推荐答案

使用以单线程方式运行 Javascript 的 node.js，如果您希望服务器最大程度地响应传入请求，那么您需要删除任何 CPU 密集型来自主要 http 服务器进程的代码.这意味着在其他一些进程中执行 CPU 密集型工作.

With node.js which runs your Javascript as single threaded, if you want your server to be maximally responsive to incoming requests, then you need to remove any CPU intensive code from the main http server process. This means doing CPU intensive work in some other process.

有很多不同的方法可以做到这一点:

There are a bunch of different approaches to doing this:

1. 使用 child_process 模块启动另一个 nodejs 应用程序，该应用程序专门用于执行 CPU 密集型工作.
2. 对您的应用进行集群，以便您拥有 N 个不同的进程，这些进程既可以执行 CPU 密集型工作，也可以处理请求.
3. 创建一个工作队列和一些工作进程来处理 CPU 密集型工作.
4. 使用较新的 Worker Threads 将 CPU 密集型工作转移到单独的 node.js 线程(需要 node v12+ 才能获得稳定的、非实验版本的线程).

1. Use the child_process module to launch another nodejs app that is purposeful built for doing your CPU intensive work.
2. Cluster your app so that you have N different processes that can do both CPU intensive work and handle requests.
3. Create a work queue and a number of worker processes that will handle the CPU intensive work.
4. Use the newer Worker Threads to move CPU intensive work to separate node.js threads (requires node v12+ for stable, non-experimental version of threads).

如果您不经常做这种 CPU 密集型工作，那么 #1 可能是最简单的.

If you don't do this CPU intensive work very often, then #1 is probably simplest.

如果您出于其他原因需要扩展(例如处理大量传入请求)并且您不经常执行 CPU 密集型工作 #2.

If you need scale for other reasons (like handling lots of incoming requests) and you don't do the CPU intensive stuff very often #2.

如果您经常执行 CPU 密集型工作，并且您希望传入的请求处理始终具有最高优先级，并且您愿意让 CPU 密集型工作花费更长的时间，那么 #3(工作队列)或 #4(线程)可能是最好的，您可以调整工作人员的数量以优化您的结果.

If you do the CPU intensive stuff pretty regularly and you want incoming request processing to always have the highest priority and you're willing to allow the CPU intensive stuff to take longer, then #3 (work queue) or #4 (threads) is probably the best and you can tune the number of workers to optimize your result.

这篇关于如何在nodejs中处理大量对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！