将生成的列表映射到celery任务的最佳方法 [英] Best way to map a generated list to a task in celery

问题描述

我正在寻找有关将一个任务生成的列表映射到芹菜另一个任务的最佳方法的建议。

I am looking for some advice as to the best way to map a list generated from a task to another task in celery.

假设我有一个名为 parse ，用于解析PDF文档并输出页面列表。然后，需要将每个页面分别传递给另一个名为 feed 的任务。所有这些都需要放入一个名为 process

Let's say I have a task called parse, which parses a PDF document and outputs a list of pages. Each page then needs to be individually passed to another task called feed. This all needs to go inside a task called process

的任务中，所以，我可以做到的一种方法是： / p>

So, one way I could do that is this:

@celery.task
def process:
    pages = parse.s(path_to_pdf).get()

    feed.map(pages)

当然，这不是好主意，因为我在任务内调用 get（）。

Of course, that is not a good idea because I am calling get() inside a task.

另外，由于我的 parse 任务包装在一个生成器函数周围，并且能够产生页面，这意味着应该有可能在产生最后一页之前将第一页排入队列

Additionally this is inefficient, since my parse task is wrapped around a generator function and is able to yield pages, which means that it should be possible to queue the first page for feeding before the last page has been yielded by the parser.

另一种可能性是这样做：

Another possibility is to do this:

@celery.task
def process:
    for page in parse.s(path_to_pdf).get():
        feed.delay(page)

该示例仍然涉及在任务中调用 get（）。另外，此示例过于简化，在添加完所有页面后（例如，在 chord 中），我确实需要做一些事情。

That example still involves calling get() inside a task though. Additionally, this example is an oversimplification and I really need to do some things after all pages have been fed (i.e. in a chord).

我正在寻找最理想的芹菜制法。

I am looking for the most optimal way to do this in celery. I would appreciate any advice.

谢谢！

推荐答案

也许对您来说太晚了，但是您可能想使用任务链：

This is probably far too late to be of use to you, but you probably want to use a task chain:

@celery.task
def process():
    return chain(parse.s(), feed_map.s())

@celery.task
def feed_map(pages):
    return feed.map(pages)

如果您有最后的任务，请说最终，您可以执行以下操作：

if you have some final task, say final, you could do this:

@celery.task
def feed_map(pages):
    return chord(feed.map.s(page) for page in pages, final.s)

这篇关于将生成的列表映射到celery任务的最佳方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！