在Celery Task Queue中，组中运行的任务与循环中的多个异步程序有什么不同吗? [英] In Celery Task Queue, is running tasks in a group any different than multiple asyncs in a loop?

问题描述

假设我有一个非常简单的任务，像这样:

Let's say I have a very simple task like this:

@celery.task(ignore_result=True)
def print_page(page):
    with open('path/to/page','w') as f:
        f.write(page)

(请忽略上面代码中的潜在竞争条件，这是一个简化的示例)

(Please ignore the potential race condition in the above code... this is a simplified example)

我的问题是，以下两个代码示例是否会产生相同的结果，或者一个示例优于另一个示例:

My question is whether the following two code samples would produce identical results, or if one is better than the other:

选择A:

@celery.task(ignore_result=True)
def print_pages(page_generator):
    for page in page_generator:
        print_page.s(page).apply_async()

选择B:

@celery.task(ignore_result=True)
def print_pages(page_generator):
    g = group(print_page.s(page) for page in page_generator)
    g.apply_async()

总的来说，我很好奇上面是否是做我正在做的事情的正确方法.本质上，我还有另一个任务，该任务分析一些数据并返回生成器，该生成器将发出文档的所有页面.对于每个页面，我想分别输出.

And in general, I am curious if the above is the correct way to do what I'm doing. Essentially, I have another task that is parses some data and returns a generator which will emit all of the pages of a document. For each page, I want to output it separately.

所以，我的链看起来像这样(也简化了):

So, my chain looks something like this (also simplified):

chain = fetch.s(url) | parse.s() | print_pages.s()
chain()

我认为，如果我能以某种方式在该链中以及该组中(实际任务之外)发出生成器，那将更有意义.但是我不确定这是可行的还是理想的.我真的很感谢您的帮助.谢谢！

I think it would make more sense if I could somehow emit the generator inside that chain and for the group there (outside of an actual task). But I am not sure if that is practical or ideal. I would really appreciate any help. Thanks!

推荐答案

您的首选似乎更好.您不希望加入扇出的print_pages任务的结果(假定ignore_result = True)，因此，一个组会增加不必要的开销/复杂性.只需像选择A中那样单独调用任务，就可以了.

Your first choice seems like the better one. You have no desire to join the results (given that ignore_result=True) of the fanned-out print_pages tasks so a group adds unnecessary overhead/complexity. Just invoke the tasks individually as in choice A and you're fine.

不过，我还要指出，Python生成器不会腌制，因此您不能将它们异步传递给Celery任务.

Further though, I'd like to note that Python generators will not pickle so you cannot pass them asynchronously to Celery tasks.

这篇关于在Celery Task Queue中，组中运行的任务与循环中的多个异步程序有什么不同吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！