使用MySQL作为作业队列 [英] Using MySQL as a job queue

问题描述

我想使用MySQL作为一个作业队列。多机将生产和消费的工作。乔布斯需要安排;有些人可能会运行每隔一小时，有的每一天，等。

I'd like to use MySQL as a job queue. Multiple machines will be producing and consuming jobs. Jobs need to be scheduled; some may run every hour, some every day, etc.

这似乎相当简单：为每个作业，有一个nextFireTime一栏，并有工人机器搜索与nextFireTime工作，更改记录为制程中的状态，然后更新nextFireTime作业时结束。

It seems fairly straightforward: for each job, have a "nextFireTime" column, and have worker machines search for the job with the nextFireTime, change the status of the record to "inProcess", and then update the nextFireTime when the job ends.

的问题是当一个工人死亡默默研究。这不会是能够更新nextFireTime或设置状态返回到空闲。

The problem comes in when a worker dies silently. It won't be able to update the nextFireTime or set the status back to "idle".

不幸的是，作业可以长时间运行，因此回收线程，看起来对已制程过长的工作是不是一种选择。有没有超时值，将工作。

Unfortunately, jobs can be long-running, so a reaper thread that looks for jobs that have been inProcess too long isn't an option. There's no timeout value that would work.

任何人都可以提出一个设计模式，将妥善处理不可靠的工人的机器？

Can anyone suggest a design pattern that would properly handle unreliable worker machines?

推荐答案

使用MySQL作为一个作业队列通常会结束痛苦，因为它是一个非常贫穷适合一个RDBMS的一般目标。用户'统'已经连接到<一个href="http://www.engineyard.com/blog/2011/5-subtle-ways-youre-using-mysql-as-a-queue-and-why-itll-bite-you/" rel="nofollow">http://www.engineyard.com/blog/2011/5-subtle-ways-youre-using-mysql-as-a-queue-and-why-itll-bite-you/,里面有很多有趣的东西说些什么。不可靠的工人仅并发症之一。

Using MySQL as a job queue generally ends in pain, as it's a very poor fit for the usual goals of an RDBMS. User 'toong' already linked to http://www.engineyard.com/blog/2011/5-subtle-ways-youre-using-mysql-as-a-queue-and-why-itll-bite-you/, which has a lot of interesting stuff to say about it. Unreliable workers are only one of the complications.

有很多，许多系统处理的工作分配，主要是由他们的队列和调度功能的复杂性区别开来。在简单的FIFO结束都像Resque，芹菜，Beanstalkd和Gearman的;在复杂的结束都像GridEngine，扭矩/毛伊岛，PBS等专业。我强烈建议新的亚马逊简单工作流系统，如果你可以容忍的亚马逊服务（我相信它的没有的要求您在EC2）。

There are many, many systems for handling job distribution, mostly distinguished by the sophistication of their queueing and scheduling capabilities. On the simple FIFO end are things like Resque, Celery, Beanstalkd, and Gearman; on the sophisticated end are things like GridEngine, Torque/Maui, and PBS Pro. I highly recommend the new Amazon Simple Workflow system, if you can tolerate reliance on an Amazon service (I believe it does not require that you be in EC2).

要你原来的问题：现在我们正在实施每个节点监控，它可以告诉，如果该节点的工作依然活跃，并发送心跳恢复到工作监视器如果是这样。这是一个痛苦，但你发现，并会继续探索，有很多的细节和错误情况进行管理。主要是，虽然，我要鼓励你做你自己一个忙，学习这个领域，并从一开始就正确地建立系统。

To your original question: right now we're implementing a per-node supervisor that can tell if the node's jobs are still active, and sending a heartbeat back to a job monitor if so. It's a pain, but as you are discovering and will continue to discover, there are a lot of details and error cases to manage. Mostly, though, I have to encourage you to do yourself a favor by learning about this domain and build the system properly from the start.

这篇关于使用MySQL作为作业队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！