Dask-如何将任务分配给特定的CPU [英] Dask - how to assign task to the specific CPU

问题描述

我正在使用Dask处理研究批，这很重(从几分钟到几个小时).任务之间没有沟通，它们只会产生副作用.我使用的机器已经虚拟化了它下面的资源(〜30个CPU)，所以我只运行 LocalCluster .有什么方法可以将特定的CPU分配给任务?在文档中，只有带有gpu和内存的示例.

I'm using Dask to process research batches, which are quite heavy (from few minutes to few hours). There's no communication between the tasks and they produce only side results. I'm using a machine which already virtualizes resources beneath it (~ 30 CPUs), so I'm just running LocalCluster. Is there any way to assign a specific CPU to a task? In docs there're only examples with gpu and memory.

我试图以类似的方式分配CPU，但是任务甚至都无法开始处理.

I've tried to assign CPU in a similar way, but the tasks won't even start to process.

client.submit(process, d, resources={'CPU': 1}) for d in data]

推荐答案

指定后任务未启动的可能原因

The likely reason that the tasks didn't start when you specified

client.submit(process, d, resources={'CPU': 1}) for d in data]

是在不指定每个工作人员都拥有该资源的情况下启动了集群(这必须在工作人员启动时完成).确保员工拥有该资源的方法如下:

is that the cluster was initiated without specifying that each worker had that resource (this has to be done at the time workers are started). Here's how to make sure that workers have that resource:

from dask.distributed import Client, LocalCluster
cluster = LocalCluster(resources={'CPU': 1})
client = Client(cluster)

要进行更细粒度的控制，可以将任务分配给特定的工作人员.首先，使用

For finer-grained control, it is possible to assign tasks to specific workers. First, get the addresses of each worker with

list_workers = list(client.scheduler_info()['workers'])

然后指定哪些工作者可以完成任务:

Then specify which worker(s) can complete the task:

# submit for completion only by the first worker in the list
results_specific_worker = [client.submit(process, d, workers=list_workers[0]) for d in data]

# submit for completion by the first two workers
results_specific_workers = [client.submit(process, d, workers=list_workers[0:2]) for d in data]

这篇关于Dask-如何将任务分配给特定的CPU的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！