用芹菜创建动态队列 [英] Create dynamic queues with Celery

问题描述

这是我的场景：

当用户登录到我的网站时，我排队给一个给定用户的一堆任务（通常每个任务需要100s的msecs每个用户有100个任务）。这些任务排队到默认的芹菜队列，我有100名工人正在运行。我使用websockets来显示用户在后台完成任务的实时进度。现在如果我有几个并发用户登录到我的网站，后面的用户排在最初的后面用户和他们的任务饿死（因为所有的任务都去了同一个队列）。我的想法是为每个用户创建一个动态队列，以确保公平。但是按照Celery文档（ http://docs.celeryproject.org /en/latest/userguide/routing.html#defining-queues ），似乎我需要静态定义队列。

关于最佳实践的任何建议为我的情况使用芹菜？

解决方案

http://docs.celeryproject.org/en/latest/userguide/workers.html#queues

  celery -A proj control add_consumer foo -d worker1.local 
  

使用app.control.add_consumer（）方法可以动态地完成相同的操作：

  app.control。 add_consumer（'foo'，reply = True）
 [{u'worker1.local'：{u'ok'：u'已经从u'foo消耗'}}] 
 
 app.control.add_consumer（'foo'，reply = Tru e，
 destination = ['[e-mail]']）
  

Here's my scenario:

When a user logs in to my website, I queue up a bunch of tasks for the given user (typically each task takes 100s of msecs and there are 100s of tasks per user). These tasks are queued to the default Celery Queue and I have 100s of workers running. I use websockets to show the user real-time progress as the tasks complete on the backend. Life is good if I have just 1 or 2 users active.

Now if I a few concurrent users log-in to my site, the latter users are queued behind the initial users and their tasks starve (since all the tasks go to the same queue). My thoughts are to create a dynamic queue per user to ensure fairness. However as per Celery documentation (http://docs.celeryproject.org/en/latest/userguide/routing.html#defining-queues), seems I need to be define the queues statically.

Any suggestions on best practices for using celery for my scenario?

解决方案

http://docs.celeryproject.org/en/latest/userguide/workers.html#queues

celery -A proj control add_consumer foo -d worker1.local

The same can be accomplished dynamically using the app.control.add_consumer() method:

app.control.add_consumer('foo', reply=True)
[{u'worker1.local': {u'ok': u"already consuming from u'foo'"}}]

app.control.add_consumer('foo', reply=True,
destination=['[e-mail]'])

这篇关于用芹菜创建动态队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！