在单独的线程NodeJ中运行长时间运行的express API进程 [英] Run long running express API process in seperate thread NodeJs

查看：154
本文介绍了在单独的线程NodeJ中运行长时间运行的express API进程的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我有一个API调用大约5-10分钟来处理。我设置了一个超时方法，以便我立即获得一个队列状态的API响应。



简单的视觉下面

  doWork（object）=> {/ * ...需要5分钟* /} 
 
 app.post（'/ longProcess'，（req，res）=> {
 setTimeout（this.doWork身体），1000）; 
 res.send（{status：'queued'}）; 
}）

这适用于第一个请求，立即响应。但是第二个请求被锁定等待doWork完成。



而不是使用SetTimeout，我真正想做的是将longProcess发送到单独的单线程



任何建议？

解决方案

h2>问题

问题不在于$ code> doWork（）需要很多时间，但是显然阻止你的线程的整个生命周期，并没有让事件循环任何机会运行。



可能的原因



这可能是由几个事情引起的，我只能在这里猜测，因为你没有显示 doWork（）的来源，甚至描述它的作用和如何。例如：




  • 您的 doWork（）可能会使用像 fs.readFileSync（）或其他功能与同步在他们的名字。

  • 您的 doWork（）可能有或的，而循环5-10分钟，并阻止事件循环。

  • 您的代码执行了一些严重的数字处理，没有分为步骤，以使事件循环在这些步骤之间滚动。 li>


通常，如果不执行，您的 doWork（）可能需要几个小时才能运行阻止你的主线程，不应该停止其他连接，甚至连毫秒都不会被送达。



解决方案



停止阻止线程



该问题的最简单的解决方案可能是避免阻塞函数调用（具有 Sync 后缀或您自己的函数），长时间运行的循环和重的计算不分为简短的步骤。



例如：




  • readFileSync（）使用 readFile（）

  • 而不是长时间运行/ while循环，使用 process.nextTick（）

  • 而不是非常深的递归（可能由于TCO），使用分为零件的循环与 process.nextTick（）



如果上述解决方案无法应用我没有办法告诉，因为我不知道你的 doWork（）函数），那么你可以采取另一种方法。还有一些其他可以做的事情。



产生一个过程



另一个解决方案是使用 child_process 可以在每次启动长时间运行的任务时生成不同的进程。当孩子结束工作并作出相应的反应时，可以通知您的主要过程，但在等待时不会被阻止。请参阅： https://nodejs.org/api/child_process.html



使用队列



您还可以使用待处理作业的队列，并通过其他进程处理它们，对主程序没有影响只会安排新任务，而不是做任务或等待他们。通常这样的队列是用Redis完成的，但也可以用CouchDB或MongoDB来完成。您需要具有一些中央注册表来处理您的工作进程可以执行的挂起任务。在Node中有很多模块要做，例如：





查看这些模块的文档，看看wh最适合您的需求。


I have an API call that takes about 5-10 minutes to process. I set a timeout method around it so that I get an immediate API response with a status of queued.

Simple visual below

doWork(object) => { /*... Takes 5 minutes */ }

app.post('/longProcess',(req,res)=> {
    setTimeout(this.doWork(req.body), 1000);
    res.send({ status: 'queued' });
})

This works for the first request giving an immediate response. But the second request is locked waiting for doWork to finish.

Instead of a using SetTimeout what I'd really like to do is send the longProcess to a separate single-thread that queues and processes theses one by one.

Any suggestions?

解决方案

The problem

The problem is not that doWork() takes a lot of time, but that it apparently blocks your thread for its entire lifetime and doesn't get the event loop any chance to run.

Probable causes

This can be caused by several things and I can only guess here since you didn't show the source of doWork() or even described what it does and how. For example:

  • Your doWork() may use blocking operations like fs.readFileSync() or other functions with Sync in their name.
  • Your doWork() may have a for or while loop that spins for 5-10 minutes and blocks the event loop while doing so.
  • Your code does some serious number crunching that is not divided into steps to let the event loop roll between those steps.

Generally, your doWork() could take hours to run if it doesn't block your main thread and it shouldn't stop other connections from getting served even for a millisecond.

Solutions

Stop blocking the thread

The simplest solution to that problem could be avoiding the blocking function calls (those with the Sync suffix or your own functions), long running loops and heavy computations that are not divided into short steps.

For examples:

  • Instead of using readFileSync() use readFile()
  • Instead of long running for/while loops, use process.nextTick()
  • Instead of very deep recursion (possibly thanks to TCO), use loops divided into parts with process.nextTick()

If the above solutions cannot be applied (which I have no way of telling since I know nothing about your doWork() function) then you may take another approach. There are some others things that you can do.

Spawn a process

Another solution would be to use child_process to spawn a different process every time you start the long running task. Your main process could be notified when the child ends doing its job and react accordingly but it wouldn't be blocked while waiting. See: https://nodejs.org/api/child_process.html

Use a queue

You can also use a queue of pending jobs and process them by other processes with no effect on your main program that would only schedule new tasks and not do them or wait for them. Usually queues like that are done with Redis but it can also be done with CouchDB or MongoDB. You need to have some central registry of pending tasks from which your worker processes could take them. There are many modules to do that in Node, for example:

See the documentation of those modules to see which one suits your needs best.

这篇关于在单独的线程NodeJ中运行长时间运行的express API进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆