使用默认计划程序进行Dask内存管理 [英] Dask Memory Management with Default Scheduler

问题描述

我一直试图在单个本地计算机上管理Dask的内存使用情况.出于某种原因，默认的Dask Client()和LocalCluster()调度程序似乎总是中断，但是Dask在不指定调度程序的情况下也能很好地工作，因此默认调度程序对我的目的是最好的，但是我发现几乎没有关于此缺省值的文档调度程序更不用说如何在其上设置RAM限制了.所有信息都是针对他们似乎无法正常工作的专用分布式客户端的.有谁知道默认调度程序文档的内容/位置，或者什至用作默认调度程序的内容(因为我什至无法在其站点/文档中找到该信息).或者，如果没有，是否有人找到了不使用Client()来限制RAM使用量的解决方案?

I have been trying to manage the memory usage of Dask on a single local machine. For some reason, the default Dask Client() and LocalCluster() scheduler always seem to break, however Dask works great without specifying the scheduler and thus the default scheduler works the best for my purposes, however I am finding almost no documentation on this default scheduler let alone how to set a RAM limit on it. All of the information is for their specialized distributed client which does not seem to work. Does anyone know what/where the default scheduler documentation can be found or what they are even using as the default scheduler (as I cannot even find that information on their site/documentation). Or if not, has anyone found a solution to limit the RAM usage without using Client()?

欢呼

推荐答案

为了控制本地dask群集中的内存限制，我使用以下设置:

in order to control the memory-limit in my local dask cluster i use the following set up:

ncores = 2
cluster = LocalCluster(n_workers=ncores, threads_per_worker=1,memory_limit =8e9)
client=Client(cluster)

这篇关于使用默认计划程序进行Dask内存管理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！