Node.js / Express和并行队列 [英] Node.js/Express and parallel queues
问题描述
我们正在建立一个具有Node.js服务器和Express的基础架构。
We are building an infrastructure which features a Node.js server and Express.
在服务器中，发生的情况如下：
In the server, what is happening is as follow:
- 服务器接受来自客户端的传入HTTP请求。
- 服务器生成两个文件（此操作可以是 ，也就是0.1秒左右）
- 服务器将生成的文件（每个〜20-200 KB）上传到外部CDN
- 服务器响应客户端，这包括CDN上的文件的URI
- The server accepts an incoming HTTP request from client.
- Server generates two files (this operation can be "relatively long", meaning also 0.1 seconds or so)
- Server uploads the generated files (~20-200 KB each) to an external CDN
- Server responds to client, and this includes the URI of the file on the CDN
目前，服务器为每个请求顺序执行此操作，相当好（Node / Express可以自动处理并发请求）。但是，当我们计划增长时，并发请求的数量可能会增加，我们认为我们最好实现一个队列来处理请求。否则，我们可能有同时运行太多任务和过多与CDN的开放连接的风险。
Currently the server is doing this sequentially for each request, and this works quite well (Node/Express can handle concurrent requests automatically). However, as we plan to grow, the number of concurrent requests may grow higher, and we believe it would be better for us to implement a queue for processing requests. Otherwise, we may risk having too many tasks running at the same time and too many open connections to the CDN. Responding to the client quickly is not a relevant thing.
我想的是在Node服务器中有一个单独的部分，包含一些
What I was thinking about is to have a separate part in the Node server that contains a few "workers" (2-3, but we will do tests to determine the correct number of simultaneous operations). So, the new flow would look something like:
- 接受来自客户端的请求后，
- 有2-3个（要测试的）工人将元素从队列中取出并执行所有操作（生成文件并将其上传到CDN）。
- 当工作者处理了操作时（如果它在队列中停留相对长的时间没关系），它通知节点服务器（回调），服务器响应
您认为这种方法是什么？你相信这是正确的吗？
What do you think of this approach? Do you believe it is the correct one?
最重要的是，如何在Node / Express中实现？
Mostly important, HOW could this be implemented in Node/Express?
谢谢您的时间
推荐答案
（回答自己的问题）
根据 Stack Overflow 上的这个问题，我的一个解决方案是使用 Caolan McMahon的异步模块。
According to this question on Stack Overflow a solution in my case would be to implement a queue using Caolan McMahon's async module.
主应用程序将创建作业和将它们推入队列，这对可以运行的并发作业数有限制。这允许同时处理任务，但是对限制进行严格控制。它在Mac OSX上像Cocoa的NSOperationQueue。
The main application will create jobs and push them into a queue, which has a limit on the number of concurrent jobs that can run. This allows processing tasks concurrently but with a strict control on the limit. It works like Cocoa's NSOperationQueue on Mac OSX.
这篇关于Node.js / Express和并行队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
