Dask Distributed - 如何为每个工作人员运行一项任务,使该任务在工作人员可用的所有内核上运行? [英] Dask Distributed - how to run one task per worker, making that task running on all cores available into the worker?

查看:62
本文介绍了Dask Distributed - 如何为每个工作人员运行一项任务,使该任务在工作人员可用的所有内核上运行?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我对使用 distributed python 库非常陌生.我有 4 个工作人员,我已经成功地为每个工作人员使用 14 个内核(在 16 个可用内核中)启动了一些并行运行,从而导致 4*14=56 个任务并行运行.

I'm very new at using distributed python library. I have 4 workers and i have successfully launched some parallel runs using 14 cores (among the 16 available) for each worker, resulting in 4*14=56 tasks running in parallel.

但是如果我只想在每个工人中一次完成一项任务,如何进行.通过这种方式,我希望在 worker 上并行使用 14 个内核来完成一项任务.

But how to proceed if I would like only one task at once in each worker. In that way, I expect one task using the 14 cores in parallel on the worker.

推荐答案

Dask Worker 维护一个单独的线程池,用于启动任务.每个任务总是从这个池中消耗一个线程.你不能告诉一个任务从这个池中获取许多线程.

Dask workers maintain a single thread pool that they use to launch tasks. Each task always consumes one thread from this pool. You can not tell a task to take many threads from this pool.

但是,还有其他方法可以控制和限制 dask 工作线程中的并发性.在您的情况下,您可以考虑定义 工人资源.这将让您停止在同一工作人员上同时运行许多大任务.

However, there are other ways to control and restrict concurrency within dask workers. In your case you might consider defining worker resources. This would let you stop many big tasks from running at the same time on the same workers.

在下面的示例中,我们定义每个工作人员都有一个 Foo 资源,并且每个任务都需要一个 Foo 来运行.这将阻止任何两个任务在同一个 worker 上同时运行.

In the following example we define that each worker has one Foo resource and that each task requires one Foo to run. This will stop any two tasks from running concurrently on the same worker.

dask-worker scheduler-address:8786 --resources Foo=1
dask-worker scheduler-address:8786 --resources Foo=1

.

from dask.distributed import Client
client = Client('scheduler-address:8786')
futures = client.map(my_expensive_function, ..., resources={'Foo': 1})

这篇关于Dask Distributed - 如何为每个工作人员运行一项任务,使该任务在工作人员可用的所有内核上运行?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆