Node.js/Express 和并行队列 [英] Node.js/Express and parallel queues
问题描述
我们正在构建一个具有 Node.js 服务器和 Express 的基础架构.
We are building an infrastructure which features a Node.js server and Express.
在服务器中，发生的情况如下:
In the server, what is happening is as follow:
- 服务器接受来自客户端的传入 HTTP 请求.
- 服务器生成两个文件(这个操作可以比较长"，也就是0.1秒左右)
- 服务器将生成的文件(每个约 20-200 KB)上传到外部 CDN
- 服务器响应客户端，其中包括 CDN 上文件的 URI
目前服务器对每个请求都按顺序执行此操作，并且效果很好(Node/Express 可以自动处理并发请求).但是，随着我们计划的增长，并发请求的数量可能会越来越高，我们认为实现一个队列来处理请求会更好.否则，我们可能会面临同时运行的任务过多以及与 CDN 的打开连接过多的风险.快速响应客户无关紧要.
Currently the server is doing this sequentially for each request, and this works quite well (Node/Express can handle concurrent requests automatically). However, as we plan to grow, the number of concurrent requests may grow higher, and we believe it would be better for us to implement a queue for processing requests. Otherwise, we may risk having too many tasks running at the same time and too many open connections to the CDN. Responding to the client quickly is not a relevant thing.
我的想法是在 Node 服务器中设置一个单独的部分，其中包含 几个worker"(2-3 个，但我们将进行测试以确定同时操作的正确数量).因此，新流程看起来像:
What I was thinking about is to have a separate part in the Node server that contains a few "workers" (2-3, but we will do tests to determine the correct number of simultaneous operations). So, the new flow would look something like:
- 服务器接受客户端的请求后，将操作添加到队列中.
- 有 2-3 个(待测试)工作人员从队列中取出元素并执行所有操作(生成文件并将它们上传到 CDN).
- 当worker处理完操作(在队列中停留的时间比较长也没关系)，通知Node服务器(回调)，服务器响应客户端(一直在等待同时).
您如何看待这种方法?你认为它是正确的吗?
What do you think of this approach? Do you believe it is the correct one?
最重要的是，这如何在 Node/Express 中实现?
Mostly important, HOW could this be implemented in Node/Express?
感谢您的宝贵时间
推荐答案
(回答我自己的问题)
根据 堆栈溢出上的这个问题，在我的情况下，解决方案是使用 Caolan McMahon 的异步模块.
According to this question on Stack Overflow a solution in my case would be to implement a queue using Caolan McMahon's async module.
主应用程序将创建作业并将它们推送到队列中，该队列对可以运行的并发作业数量有限制.这允许同时处理任务，但对限制进行严格控制.它的工作方式类似于 Mac OSX 上的 Cocoa 的 NSOperationQueue.
The main application will create jobs and push them into a queue, which has a limit on the number of concurrent jobs that can run. This allows processing tasks concurrently but with a strict control on the limit. It works like Cocoa's NSOperationQueue on Mac OSX.
这篇关于Node.js/Express 和并行队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
