忽略Dask分布式中长时间运行的任务 [英] Ignore long running tasks in Dask distributed

问题描述

在这篇文章中讨论了已经开始的任务无法在Dask中取消(语言限制).

In this post is discussed that already started tasks can not be canceled in Dask (language limitation).

但是，如果我只想省略那些任务怎么办?

But what if I just want to omit those tasks?

start_computing_time = time.time()

for future in task_pool:
    if condition:
       do_something_long(future.result()) 
    else:       
       future.cancel()

total_computing_time = time.time() - start_computing_time

在我的应用程序中，执行时间至关重要.满足停止条件后，我只想省略正在运行的任务，因为我不再对这些结果感兴趣.据我所知， future.cancel()只会取消尚未运行的期货.

In my application, execution time is critical. Once the stopping condition is met, I just want to omit running tasks, as I am no longer interested on those results. As to my knowledge, future.cancel() will just cancel not yet running futures.

但是对于那些正在执行的任务，有什么办法可以忽略它们?

But for those tasks in execution, is there any way to ignore them?

提前谢谢！

推荐答案

听起来像 as_completed 迭代器可能会解决您的问题.您可以等待一组期货，并在它们到达时更新系统.然后，您有足够的信息(或超时已过去)，只需继续操作并删除运行中的期货即可.

It sounds like the as_completed iterator might solve your problem. You can wait on a set of futures and update your system as they arrive. Then you have enough information (or a timeout has passed) you would just move on and delete the running futures.

这篇关于忽略Dask分布式中长时间运行的任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！