是否可以将np数组用作h5py数据集中的索引? [英] is it possible to use np arrays as indices in h5py datasets?

问题描述

我需要将多个数据集(每个都包含在一个单独的文件中)合并到另一个属于最终文件的数据集中. 当部分数据集中的数据被复制到最终数据中时，将不会保留这些数据的顺序-通过索引将部分数据集中的数据映射"到最终的数据中.我创建了两个列表，final_indices和partial_indices，并写道:

I need to merge a number of datasets, each contained in a separate file, into another dataset belonging to a final file. The order of the data in the partial dataset is not preserved when they get copied in the final one - the data in the partial datasets is 'mapped' into the final one through indices. I created two lists, final_indices and partial_indices, and wrote:

final_dataset   = final_hdf5file['dataset']
partial_dataset = partial_hdf5file['dataset']

# here partial ad final_indices are lists.
final_dataset[final_indices] = partial_dataset[partial_indices] 

问题是性能很差-原因是final_和partial_indices都必须列出. 我的解决方法是从最终和部分数据集中创建两个np数组，并使用np数组作为索引.

the problem with this is that the performance is quite bad - and the reason is that final_ and partial_indices have both to be lists. my workaround has been to create two np arrays from the final and partial datasets, and use np arrays as indices.

final_array   = np.array(final_dataset)
partial_array = np.array(partial_dataset)
# here partial ad final_indices are nd arrays.
final_array[final_indices] = partial_array[partial_indices] 

然后将最终数组重写为最终数据集.

The final array is then re-written to the final dataset.

final_dataset[...] = final_array

但是，在我看来，这样做并不明智.

However, it seems to me rather inelegant to do so.

是否可以将np.arrays用作h5py数据集中的索引?

Is it possible to use np.arrays as indices in a h5py dataset?

推荐答案

因此，您正在为读取和写入做花式索引:

So you are doing fancy-indexing for both the read and write:

http://docs.h5py.org/en /latest/high/dataset.html#fancy-indexing

它警告说，如果列表较长，它可能会变慢.

It warns that it can be slow with long lists.

我可以看到在哪里读写整个集合，以及在数组上进行映射会更快，尽管我实际上并未对此进行测试.读/写速度更快，映射也是如此

I can see where reading and writing the whole sets, and doing the mapping on arrays will be faster, though I haven't actually tested that. The read/writing is faster, as is the mapping

http://docs.h5py.org /en/latest/high/dataset.html#reading-writing-data

我将使用切片符号(或value)加载数据集，但这只是次要点.

I would use the slice notation (or value) to load the datasets, but that's a minor point.

final_array   = final_dataset[:]

将代码隐藏在函数中(如果看起来不太美观).

Hide the code in a function if it looks inelegant.

此oneliner可能有效(我尚未测试过). RHS更可能起作用.

This oneliner might work (I haven't tested it). The RHS is more likely to work.

final_dataset[:][final_indices] = partial_dataset[:][partial_indices] 

这篇关于是否可以将np数组用作h5py数据集中的索引?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！