python executor从完成的回调中产生任务(递归提交任务) [英] python executor spawn tasks from done callback (recursively submit tasks)

问题描述

我正在尝试根据已完成任务的结果提交更多任务:

I'm trying to submit further tasks from result of a task that was done:

with concurrent.futures.ThreadPoolExecutor() as executor:
    future = executor.submit(my_task)
    def callback(future):
        for another_task in future.result():
            future = executor.submit(another_task)
            future.add_done_callback(callback)
    future.add_done_callback(callback)

但我得到:

运行时错误:关闭后无法安排新的期货

RuntimeError: cannot schedule new futures after shutdown

让执行者保持回调的最佳方法是什么?信号量?

What's the best way to make the executor hold for the callback? A semaphore?

理想情况下，如果将 ThreadPoolExecutor 替换为 ProcessPoolExecutor，解决方案应该可以很好地传输.

Ideally the solution should transfer well if ThreadPoolExecutor is replaced with ProcessPoolExecutor.

推荐答案

此解决方案可保证在任何给定点最大化处理.sleep 的使用不是那么优雅，但这是我迄今为止最好的.

This solution is guaranteed to max out processing at any given point. The use of sleep is not that elegant but this is the best I have so far.

with concurrent.futures.ThreadPoolExecutor() as executor:
    pending_tasks = 1
    future = executor.submit(my_task)
    def callback(future):
        nonlocal pending_tasks
        for another_task in future.result():
            pending_tasks += 1
            future = executor.submit(another_task)
            future.add_done_callback(callback)
        pending_tasks -= 1
    future.add_done_callback(callback)
    while pending_tasks:
        time.sleep(10)

这篇关于python executor从完成的回调中产生任务(递归提交任务)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！