使用Dask访问大型已发布数组中的单个元素 [英] Access a single element in large published array with Dask

问题描述

是否有一种更快的方法，可以使用Dask仅检索大型已发布数组中的单个元素，而无需检索整个数组？

在下面的示例client.get_dataset（'array1'）[0]与client.get_dataset（'array1'）花费的时间大致相同。

In the example below client.get_dataset('array1')[0] takes roughly the same time as client.get_dataset('array1').

import distributed
client = distributed.Client()
data = [1]*10000000
payload = {'array1': data}
client.publish(**payload)

one_element = client.get_dataset('array1')[0]

推荐答案

请注意，您发布的所有内容都将发送给调度程序，而不是工作人员，因此这效率很低。 Publish打算与dask集合（例如dask.array）一起使用。

Note that anything you publish goes to the scheduler, not to the workers, so this is somewhat inefficient. Publish was intended to be used with Dask collections like dask.array.

import dask.array as da
x = da.ones(10000000, chunks=(100000,))  # 1e7 size array cut into 1e5 size chunks
x = x.persist()  # persist array on the workers of the cluster

client.publish(x=x)  # store the metadata of x on the scheduler

Client 2

Client 2

x = client.get_dataset('x')  # get the lazy collection x
x[0].compute()  # this selection happens on the worker, only the result comes down

这篇关于使用Dask访问大型已发布数组中的单个元素的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！