使用Dask计算导致执行挂起 [英] Using Dask compute causes execution to hang

问题描述

这是对我以前关于使用Dask计算的以下问题之一的潜在答案的后续问题：访问大型数组中的一个元素。

This is a follow up question to a potential answer to one of my previous questions on using Dask computed to access one element in a large array .

为什么使用Dask计算会导致执行挂起？
这是工作代码段：

Why does using Dask compute cause the execution to hang below? Here's the working code snippet:

#Suppose you created a scheduler at the ip address of 111.111.11.11:8786


from dask.distributed import Client
import dask.array as da

# client1
client1 = Client("111.111.11.11:8786")
x = da.ones(10000000, chunks=(100000,))  # 1e7 size array cut into 1e5 size chunks
x = x.persist()
client1.publish_dataset(x=x)

# client2
client2 = Client("111.111.11.11:8786")
x = client2.get_dataset('x')  #get the lazy collection x
result = x[0].compute() #code execution hangs here
print(result)

推荐答案

persist 的行为有所不同，具体取决于您是否有处于活动状态的分布式客户端。在您的情况下，您在建立任何客户端之前先调用它，结果所有数据都打包在图形描述中。在线程调度程序上，此行为是可以的，在该线程上，工作人员之间共享内存，但是在发布时，您会将整个内容发送给调度程序，显然很令人窒息。

persist behaves differently, depending on whether you have a distributed client active or not. In your case, you call it before making any client, with the result that the whole of the data is packed into the graph description. This behaviour is OK on the threaded scheduler, where memory is shared between workers, but when you publish, you are sending the whole thing to the scheduler, and apparently it is choking.

如果首先创建 client1 ，您会注意到持久性发生得非常快（在这种情况下，调度程序仅获取指向数据的指针），以及发布-获取周期将按预期工作。

If you make client1 first, you will notice that persist happens very quickly (the scheduler is only getting pointers to the data in this case), and the publish-fetch cycle will work as expected.

这篇关于使用Dask计算导致执行挂起的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！