流程异步spawing：设计问题 - 芹菜或扭曲 [英] Async spawing of processes: design question - Celery or Twisted

问题描述

所有：我正在寻找输入/引导/和设计思路。我的目标是要找到一个精干而可靠的方式，采取从HTTP POST XML有效载荷（本部分没有问题），分析它，并异步产生一个相对长寿命的过程。

All: I'm seeking input/guidance/and design ideas. My goal is to find a lean but reliable way to take XML payload from an HTTP POST (no problems with this part), parse it, and spawn a relatively long-lived process asynchronously.

衍生的进程是CPU密集型的，将持续的大约三分钟。我不奢望太多负荷在第一，但有我需要在服务器水平扩展了这一点，如交通有望增加一定的可能性。

The spawned process is CPU intensive and will last for roughly three minutes. I don't expect much load at first, but there's a definite possibility that I will need to scale this out horizontally across servers as traffic hopefully increases.

我真的很喜欢芹菜/ Django的堆栈用于该用途：它是非常直观，并拥有所有内置的框架来完成正是我需要的。我开始来与热情这条道路，但我很快就发现我的小512MB RAM的云服务器只有空闲内存100MB，我开始感觉到，我的标题是麻烦，一旦我与所有正在运行的全速我的流程去住。此外，它有几个运动部件：RabbitMQ的，MySQL和cerleryd，ligthttpd和Django的容器

I really like the Celery/Django stack for this use: it's very intuitive and has all of the built-in framework to accomplish exactly what I need. I started down that path with zeal, but I soon found my little 512MB RAM cloud server had only 100MB of free memory and I started sensing that I was headed for trouble once I went live with all of my processes running full-tilt. Also, it's got several moving parts: RabbitMQ, MySQL, cerleryd, ligthttpd and the django container.

我绝对可以增加我的服务器的大小，但我希望让我的成本降低到最低限度在这个项目的早期阶段。

I can absolutely increase the size of my server, but I'm hoping to keep my costs down to a minimum at this early phase of this project.

作为替代方案，我考虑使用双绞线用于过程管理，以及对远程系统的角度经纪人，应需要它们。但至少对我来说，当扭曲是辉煌的，我觉得我报名参加了很多走这路：写协议，回调管理，跟踪工作状态等。这里的好处是pretty明显 - 出色的性能，更少的运动部件，和一个更小的内存占用（注：我需要验证内存部分）。我大量的Python朝倾斜这 - 这对我来说更愉快比选择：）

As an alternative, I'm considering using twisted for the process management, as well as perspective broker for the remote systems, should they be needed. But for me at least, while twisted is brilliant, I feel like I'm signing up for a lot going down that path: writing protocols, callback management, keeping track of job states, etc. The benefits here are pretty obvious - excellent performance, far fewer moving parts, and a smaller memory footprint (note: I need to verify the memory part). I'm heavily skewed toward Python for this - it's much more enjoyable for me than the alternatives :)

我会非常AP preciate有这方面的观点。我担心在错误的轨道上开始做事了，和重做稍后与生产业务将是痛苦的。

I'd greatly appreciate any perspective on this. I'm concerned about starting things off on the wrong track, and redoing this later with production traffic will be painful.

-Matt

推荐答案

在我的系统，用的RabbitMQ pretty合理的默认运行使用有关RAM 2MB。 Celeryd使用多一点，但不能过量。

On my system, RabbitMQ running with pretty reasonable defaults is using about 2MB of RAM. Celeryd uses a bit more, but not an excessive amount.

在我看来，RabbitMQ的开销和芹菜pretty多相比可以忽略不计到堆栈的其余部分。如果你正在处理的要采取几分钟才能完成的工作，这些工作是什么会尽快压倒你的512MB服务器作为流量的增加，而不是RabbitMQ的。与RabbitMQ的芹菜出发至少会设置你很好地扩展这些工作了水平的，所以你在正确的轨道上有肯定是

In my opinion, the overhead of RabbitMQ and celery are pretty much negligible compared to the rest of the stack. If you're processing jobs that are going to take several minutes to complete, those jobs are what will overwhelm your 512MB server as soon as your traffic increases, not RabbitMQ. Starting off with RabbitMQ and Celery will at least set you up nicely to scale those jobs out horizontally though, so you're definitely on the right track there.

当然，你可以写在扭曲自己的作业控制，但我没有看到它获得你多少。双绞线有pretty不错的表现，但我不希望它被足够胜过的RabbitMQ来证明引入bug和体系结构限制时间和潜力。大多数情况下，它似乎只是错误的地方发愁优化。就拿你会已经花了重新编写的RabbitMQ和工作在20％或东西减少这些3分钟工作的时间。或者只是花费额外的$ 20 /月，增加一倍，你的能力。

Sure, you could write your own job control in Twisted, but I don't see it gaining you much. Twisted has pretty good performance, but I wouldn't expect it to outperform RabbitMQ by enough to justify the time and potential for introducing bugs and architectural limitations. Mostly, it just seems like the wrong spot to worry about optimizing. Take the time that you would've spent re-writing RabbitMQ and work on reducing those three minute jobs by 20% or something. Or just spend an extra $20/month and double your capacity.

这篇关于流程异步spawing：设计问题 - 芹菜或扭曲的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！