Web 应用程序架构 - 需要作业/任务队列吗? [英] Web Application Architecture - Job/Task Queue needed?

问题描述

我目前正在设计一个 Web 应用程序，该应用程序将允许用户安排将针对 HTTP API(代表他们)执行的任务.任务可以重复执行，可用于调度的最小时间分辨率为一分钟.由于任务的性质，我认为异步执行它们是有意义的.但是，这部分的架构应该是怎样的?

I am currently designing a web application that will allow users to schedule tasks which will be executed against an HTTP API (on behalf of them). The tasks can be recurring and the minimal time resolution that can be used for scheduling will be one minute. Because of the nature of the tasks I think it makes sense to execute them asynchronously. However, how should the architecture of this part look like?

我考虑过使用任务队列由 Web 应用程序创建任务并让它们由工作人员执行.在这种情况下，我有几个问题:

I thought about using a task queue to create tasks by the web application and let them be executed by a worker. In this case, I have several questions:

• 我如何处理重复性任务?
• 如何轻松保存任务结果?
• 是否可以轻松地使队列持久化"?
• 工作人员是否应该直接与数据库交互?
• 我应该手动将重复性任务排队吗?

我还能考虑什么?由于我认为我不是唯一一个考虑这种 Web 应用程序架构的人，是否有任何最佳实践"?任务队列是要走的路吗?

What else could I consider? Since I assume I am not the only one thinking about this kind of web application architecture, are there any "best practices"? Is a task queue the way to go?

推荐答案

是的，这是一个众所周知的模式，用于在 Web 应用程序的后端处理长期存在的任务.根据您的语言和应用程序框架，有许多队列实现 - 例如Resque 或 Beanstalkd 或 ActiveMQa> 或者如果你的性能要求不高，你可以使用数据库表作为一种队列.

Yes this is a well-known pattern for handling long-lived tasks at the back end of a web application. Depending on your langauge and application framework there are a number of queue implementations out there - e.g. Resque or Beanstalkd or ActiveMQ or if your performance requirements are not high you can use a database table as a kind of queue.

基本思想是您的 Web 应用程序将作业放置在一个队列中，该队列具有足够的内容以使作业能够继续进行.后台的一组工作进程(理想情况下独立于您的 Web 应用程序运行)从队列中读取作业并执行它们.结果可以写回回复队列或写入数据库.这取决于您希望如何向用户显示结果.对于 Web 应用程序，将结果写入数据库可能更有意义.

The basic idea is your Web application places jobs on a queue with enough content to enable to job to proceed. A group of worker processes in the background (ideally running independent of your web application) read the jobs off the queue and execute them. The results can be written back onto a reply queue or perhaps written into a database. It depends on how you want to display the results back to the user. For a web application, writing results into the database probably makse more sense.

根据您的队列处理程序，它们可以使作业持久化.例如.ActiveMQ 支持持久性消息传递，以便在发生故障时恢复队列中的消息.

Depending on your queue handler, they can make jobs persistent. E.g. ActiveMQ supports persistent messaging so that messages on a queue are recovered in the event of a failure.

您询问重复性工作 - 我认为答案取决于它们何时需要重复.

You ask about recurring jobs - and I think the answer depends on when they need to recur.

直接消息队列将在消息可用时立即处理/发布消息给工作人员.所以调度是棘手的或不可能的.为了支持计划作业(包括在给定时间或一段时间后重复出现的作业)，您可能应该将数据库表用作具有开始时间"属性的简单队列.

A straight message queue will process/release messages to workers as soon as they become available. So scheduling is tricky or impossible. To support scheduled jobs (including jobs which recur at a given time or after a time elapse) you should probably look at using a database table as a simple queue with a "start time" attribute.

我最近在此处描述了类似的模式.

I recently described a similar pattern here.

这篇关于Web 应用程序架构 - 需要作业/任务队列吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！