如何将Linux cron作业转换为“亚马逊方式"? [英] How to convert Linux cron jobs to "the Amazon way"?

问题描述

无论是好是坏，我们已经将整个 LAMP 网络应用程序从专用计算机迁移到了云端( Amazon EC2计算机).到目前为止，一切都很好，但是我们 crons 的方式并不理想.我有一个特定于亚马逊的问题，关于如何使用亚马逊方式"最好地管理云中的cron作业.

For better or worse, we have migrated our whole LAMP web application from dedicated machines to the cloud (Amazon EC2 machines). It's going great so far but the way we do crons is sub-optimal. I have a Amazon-specific question about how to best manage cron jobs in the cloud using "the Amazon way".

问题:我们有多个网络服务器，并且需要为批处理作业(例如，创建RSS提要，触发电子邮件，实际上有许多其他事情)运行cron.但是cron作业只需要在一台机器上运行，因为它们经常写入数据库，因此如果在多台机器上运行，将复制结果.

The problem: We have multiple webservers, and need to run crons for batch jobs such as creating RSS feeds, triggering emails, many different things actually. BUT the cron jobs need to only run on one machine because they often write to the database so would duplicate the results if run on multiple machines.

到目前为止，我们将其中一个Web服务器指定为主Web服务器"，它具有其他Web服务器所没有的一些特殊"任务.云计算的权衡是可靠性-我们不希望有主服务器"，因为它是单点故障.我们希望它们全部相同，并且能够进行升迁和降尺度，而又不记得不要将主Web服务器从群集中移出.

So far, we designated one of the webservers as the "master-webserver" and it has a few "special" tasks that the other webservers don't have. The trade-off for cloud computing is reliability - we don't want a "master-webserver" because it's a single point of failure. We want them to all be identical and to be able to upscale and downscale without remembering not to take the master-webserver out of the cluster.

我们如何重新设计应用程序以将Linux cron作业转换为没有单点故障的临时工作项?

How can we redesign our application to convert Linux cron jobs into transitory work items that don't have a single point of failure?

到目前为止我的想法:

• 有一台专用于仅运行克朗的机器.这将更易于管理，但仍将是单点故障，并且由于额外的实例而浪费一些钱.
• 可以想象有些工作可以从Linux分支转移到 MySQL事件，但是我我不太喜欢这个想法，因为我不想将应用程序逻辑放入数据库层.
• 也许我们可以在所有计算机上运行所有cron，但是可以更改cron脚本，因此它们都以实现锁定机制的逻辑开头，因此只有一台服务器实际采取了行动，而其他服务器只是跳过了.我不喜欢这个想法，因为它听起来可能有问题，我宁愿使用亚马逊的最佳实践，而不是自己动手做.
• 我正在想象这样一种情况:将作业安排在某个地方，添加到队列中，然后Web服务器可能每个都是工人，可以说嘿，我来接这个". Amazon Simple Workflow Service 听起来完全是这种事情，但是我目前对此了解不多，所以任何细节都将有所帮助.像cron这样简单的东西似乎有点沉重?是正确的服务还是更合适的亚马逊服务?

• Have a machine dedicated to only running crons. This would be a little more manageable but would still be a single-point-of-failure, and would waste some money having an extra instance.
• Some jobs could conceivably be moved from Linux crons to MySQL Events however I'm not a big fan of this idea as I don't want to put application logic into the database layer.
• Perhaps we can run all crons on all machines but change our cron scripts so they all start with a bit of logic that implements a locking mechanism so only one server actually takes action and the others just skip. I'm not a fan of this idea as it sounds potentially buggy and I would prefer to use a Amazon best-practice rather than rolling our own.
• I'm imagining a situation where jobs are scheduled somewhere, added to a queue and then the webservers could each be a worker, that can say "hey, I'll take this one". Amazon Simple Workflow Service sounds exactly this kind of thing but I don't currently know much about it so any specifics would be helpful. It seems kind of heavy-weight for something as simple as a cron? Is it the right service or is there a more suitable Amazon service?

更新:自问了这个问题以来，我已经看过 Amazon Simple YouTube上的工作流服务网络研讨会，并在34:40受到关注( http://www.youtube. com/watch?v = lBUQiek8Jqk#t = 34m40s )，我看到了一张幻灯片，其中提到了cron作业作为示例应用程序.在他们的文档页面"适用于Amazon SWF的AWS Flow Framework示例"中，亚马逊表示他们具有cron的示例代码:

Update: Since asking the question I have watched the Amazon Simple Workflow Service webinar on YouTube and noticed at 34:40 (http://www.youtube.com/watch?v=lBUQiek8Jqk#t=34m40s) I caught a glimpse of a slide mentioning cron jobs as a sample application. In their documentation page, "AWS Flow Framework samples for Amazon SWF", Amazon say they have sample code for crons:

... > Cron作业在此示例中，长期运行的工作流会定期 执行一项活动.能够像新人一样继续执行 执行，以便执行可以在很长一段时间内运行 时间被证明. ...

... > Cron jobs In this sample, a long running workflow periodically executes an activity. The ability to continue executions as new executions so that an execution can run for very extended periods of time is demonstrated. ...

我下载了适用于Java的AWS开发工具包( http://aws.amazon.com/sdkforjava/)，而且确实足够埋在一个荒谬的文件夹层中，其中有一些Java代码(aws-java-sdk-1.3.6/samples/AwsFlowFramework/src/com/amazonaws/services/simpleworkflow/flow/examples/periodicworkflow).

I downloaded the AWS SDK for Java (http://aws.amazon.com/sdkforjava/) and sure enough buried within a ridiculous layers of folders there is some java code (aws-java-sdk-1.3.6/samples/AwsFlowFramework/src/com/amazonaws/services/simpleworkflow/flow/examples/periodicworkflow).

问题是，老实说，这并没有真正的帮助，因为这不是我可以轻松利用自己的技能来消化的东西. PHP SDK中缺少相同的示例，而且似乎没有教程可循此过程.因此，基本上，我仍在寻找建议或技巧.

The problem is, if I'm honest, this doesn't really help as it's not something I can easily digest with my skillset. The same sample is missing from the PHP SDK and there doesn't seem to be a tutorial that walks though the process. So basically, I'm still hunting for advice or tips.

推荐答案

16年2月12日，亚马逊在博客上发布了

On 12/Feb/16 Amazon blogged about Scheduling SSH jobs using AWS Lambda. I think this answers the question.

这篇关于如何将Linux cron作业转换为“亚马逊方式"?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！