Web应用程序体系结构 - 作业/任务队列需要的？ [英] Web Application Architecture - Job/Task Queue needed?

问题描述

我目前正在设计一个Web应用程序，允许用户安排将针对HTTP API来执行（代表他们的）任务。任务可以是周期性以及可用于调度的最小时间分辨率将是一分钟。由于任务的性质，我觉得很有道理异步执行它们。然而，这部分的建筑应该如何样子？

我想过使用任务队列创建的Web应用程序的任务，让他们由工人执行。在这种情况下，我有几个问题：

• 如何处理重复的任务？


• 如何轻松保存的任务的结果？


• 是很容易可以使队列老大难？


• 如若工人直接与数据库进行交互？


• 我应该排队重复任务手动的？

还有什么我可以考虑？因为我认为我不是唯一一个想这种Web应用程序体系结构，是否有任何最佳做法？是一个任务队列的路要走？

解决方案

是的，这是在一个Web应用程序的后端处理长期存在的任务知名的格局。根据您的langauge和应用程序框架也有一些队列实现在那里 - 例如 Resque 或的 Beanstalkd 或的ActiveMQ 或如果你的性能要求并不高，你可以使用一个数据库表作为一种队列。

其基本思想是与足够的内容队列Web应用程序的地方工作，以使对工作继续进行。在后台一组工作进程（理想情况下独立运行的Web应用程序）的读出队列中的作业并执行它们。结果能够写回到一个应答队列或者写入到数据库中。这取决于你想如何显示结果反馈给用户。对于Web应用程序，写结果到数据库可能makse更有意义。

根据您的队列处理程序，他们可以持久的工作。例如。 ActiveMQ的支持持久消息使得在一队列中的消息在出现故障的情况下被回收。

您问的重复性作业 - 我认为，答案取决于当他们需要对复发

。

一个笔直的消息队列会尽快变得可用处理/释放消息给工人。因此调度是棘手的或不可能的。为了支持调度作业（包括其再次发生在特定的时间或经过时间的流逝作业）你应该看看使用数据库表作为一个简单的队列一个开始时间属性。

我最近描述了类似的模式<一个href=\"http://stackoverflow.com/questions/5835565/activerecord-handling-db-races-among-workers\">here.

I am currently designing a web application that will allow users to schedule tasks which will be executed against an HTTP API (on behalf of them). The tasks can be recurring and the minimal time resolution that can be used for scheduling will be one minute. Because of the nature of the tasks I think it makes sense to execute them asynchronously. However, how should the architecture of this part look like?

I thought about using a task queue to create tasks by the web application and let them be executed by a worker. In this case, I have several questions:

• How do I handle recurring tasks?
• How do I easily save the results of the tasks?
• Is it easily possible to make the queue "persistent"?
• Should the workers directly interact with a database?
• Should I queue recurring tasks manually up?

What else could I consider? Since I assume I am not the only one thinking about this kind of web application architecture, are there any "best practices"? Is a task queue the way to go?

解决方案

Yes this is a well-known pattern for handling long-lived tasks at the back end of a web application. Depending on your langauge and application framework there are a number of queue implementations out there - e.g. Resque or Beanstalkd or ActiveMQ or if your performance requirements are not high you can use a database table as a kind of queue.

The basic idea is your Web application places jobs on a queue with enough content to enable to job to proceed. A group of worker processes in the background (ideally running independent of your web application) read the jobs off the queue and execute them. The results can be written back onto a reply queue or perhaps written into a database. It depends on how you want to display the results back to the user. For a web application, writing results into the database probably makse more sense.

Depending on your queue handler, they can make jobs persistent. E.g. ActiveMQ supports persistent messaging so that messages on a queue are recovered in the event of a failure.

You ask about recurring jobs - and I think the answer depends on when they need to recur.

A straight message queue will process/release messages to workers as soon as they become available. So scheduling is tricky or impossible. To support scheduled jobs (including jobs which recur at a given time or after a time elapse) you should probably look at using a database table as a simple queue with a "start time" attribute.

I recently described a similar pattern here.

这篇关于Web应用程序体系结构 - 作业/任务队列需要的？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！