如何将数组存储在无法加载到内存的hdf5文件中? [英] How to store an array in hdf5 file which is too big to load in memory?

问题描述

有什么方法可以将数组存储在hdf5文件中，该文件太大而无法加载到内存中?

Is there any way to store an array in an hdf5 file, which is too big to load in memory?

如果我做这样的事情

f = h5py.File('test.hdf5','w')
f['mydata'] = np.zeros(2**32)

我遇到内存错误.

推荐答案

根据文档，您可以使用create_dataset创建存储在hdf5中的分块数组.示例:

According to the documentation, you can use create_dataset to create a chunked array stored in the hdf5. Example:

>>> import h5py
>>> f = h5py.File('test.h5', 'w')
>>> arr = f.create_dataset('mydata', (2**32,), chunks=True)
>>> arr
<HDF5 dataset "mydata": shape (4294967296,), type "<f4">

切片HDF5 dataset返回Numpy数组.

Slicing the HDF5 dataset returns Numpy-arrays.

>>> arr[:10]
array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.], dtype=float32)
>>> type(arr[:10])
numpy.array

您可以为Numpy数组设置值.

You can set values as for a Numpy-array.

>>> arr[3:5] = 3
>>> arr[:6]
array([ 0.,  0.,  0.,  3.,  3.,  0.], dtype=float32)

我不知道这是否是最有效的方法，但是您可以分块迭代整个数组.例如，将其设置为随机值:

I don't know if this is the most efficient way, but you can iterate over the whole array in chunks. And for instance setting it to random values:

>>> import numpy as np
>>> for i in range(0, arr.size, arr.chunks[0]):
        arr[i: i+arr.chunks[0]] = np.random.randn(arr.chunks[0])
>>> arr[:5]
array([ 0.62833798,  0.03631227,  2.00691652, -0.16631022,  0.07727782], dtype=float32)

这篇关于如何将数组存储在无法加载到内存的hdf5文件中?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！