h5py-将对象动态写入文件吗? [英] h5py - Write object dynamically to file?
问题描述
我正在尝试将常规的python对象(几个键/值对)写入hdf5文件.我正在将h5py 2.7.0与python 3.5.2.3一起使用.
I am trying to write regular python objects (which several key/value pairs) to a hdf5 file. I am using h5py 2.7.0 with python 3.5.2.3.
现在,我正在尝试将一个对象完整地写入数据集中:
Right now, I am trying to write one object in its entirety to a dataset:
#...read dataset, store one data object in 'obj'
#obj could be something like: {'value1': 0.09, 'state': {'angle_rad': 0.034903, 'value2': 0.83322}, 'value3': 0.3}
dataset = h5File.create_dataset('grp2/ds3', data=obj)
这会产生错误,因为基础dtype
无法转换为native HDF5 equivalent
:
This produces an error as the underlying dtype
can not be converted to a native HDF5 equivalent
:
File "\python-3.5.2.amd64\lib\site-packages\h5py\_hl\group.py", line 106, in create_dataset
dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
File "\python-3.5.2.amd64\lib\site-packages\h5py\_hl\dataset.py", line 100, in make_new_dset
tid = h5t.py_create(dtype, logical=1)
File "h5py\h5t.pyx", line 1543, in h5py.h5t.py_create (D:\Build\h5py\h5py-hdf5
110-git\h5py\h5t.c:18116)
File "h5py\h5t.pyx", line 1565, in h5py.h5t.py_create (D:\Build\h5py\h5py-hdf5
110-git\h5py\h5t.c:17936)
File "h5py\h5t.pyx", line 1620, in h5py.h5t.py_create (D:\Build\h5py\h5py-hdf5
110-git\h5py\h5t.c:17837)
TypeError: Object dtype dtype('O') has no native HDF5 equivalent
是否可以以动态"方式将对象写入HDF5文件?
Is it possible to write the object to a HDF5 file in a "dynamic" way?
推荐答案
如果要保存的对象是带有数字值的嵌套字典,则可以使用H5文件的group/set
结构重新创建.
If the object you want save is a nested dictionary, with numeric values, then it could be recreated with the group/set
structure of a H5 file.
一个简单的递归函数将是:
A simple recursive function would be:
def write_layer(gp, adict):
for k,v in adict.items():
if isinstance(v, dict):
gp1 = gp.create_group(k)
write_layer(gp1, v)
else:
gp.create_dataset(k, data=np.atleast_1d(v))
In [205]: dd = {'value1': 0.09, 'state': {'angle_rad': 0.034903, 'value2': 0.83322}, 'value3': 0.3}
In [206]: f = h5py.File('test.h5', 'w')
In [207]: write_layer(f, dd)
In [208]: list(f.keys())
Out[208]: ['state', 'value1', 'value3']
In [209]: f['value1'][:]
Out[209]: array([ 0.09])
In [210]: f['state']['value2'][:]
Out[210]: array([ 0.83322])
您可能希望对其进行优化,然后将标量保存为属性而不是完整的数据集.
You might want to refine it and save scalars as attributes rather full datasets.
def write_layer1(gp, adict):
for k,v in adict.items():
if isinstance(v, dict):
gp1 = gp.create_group(k)
write_layer1(gp1, v)
else:
if isinstance(v, (np.ndarray, list)):
gp.create_dataset(k, np.atleast_1d(v))
else:
gp.attrs.create(k,v)
In [215]: list(f.keys())
Out[215]: ['state']
In [218]: list(f.attrs.items())
Out[218]: [('value3', 0.29999999999999999), ('value1', 0.089999999999999997)]
In [219]: f['state']
Out[219]: <HDF5 group "/state" (0 members)>
In [220]: list(f['state'].attrs.items())
Out[220]: [('value2', 0.83321999999999996), ('angle_rad', 0.034903000000000003)]
检索数据集和属性的混合更为复杂,尽管您可以编写代码将其隐藏.
Retrieving the mix of datasets and attributes is more complicated, though you could write code to hide that.
这是一种结构化数组方法(具有复合dtype)
Here's a structured array approach (with a compound dtype)
定义一个与字典结构匹配的dtype.这样的嵌套是可能的,但如果嵌套太深,则可能会很尴尬:
Define a dtype that matches your dictionary structure. Nesting like this is possible, but can be awkward if too deep:
In [226]: dt=[('state',[('angle_rad','f'),('value2','f')]),
('value1','f'),
('value3','f')]
In [227]: dt = np.dtype(dt)
使用多个记录制作这种类型的空白数组;用您的词典中的数据填写一条记录.注意,元组的嵌套必须与dtype嵌套匹配.通常将结构化数据作为此类元组的列表提供.
Make a blank array of this type, with several records; fill in one record with data from your dictionary. Note that the nest of tuples has to match the dtype nesting. More generally structured data is provided as a list of such tuples.
In [228]: arr = np.ones((3,), dtype=dt)
In [229]: arr[0]=((.034903, 0.83322), 0.09, 0.3)
In [230]: arr
Out[230]:
array([(( 0.034903, 0.83322001), 0.09, 0.30000001),
(( 1. , 1. ), 1. , 1. ),
(( 1. , 1. ), 1. , 1. )],
dtype=[('state', [('angle_rad', '<f4'), ('value2', '<f4')]), ('value1', '<f4'), ('value3', '<f4')])
直接将数组写入h5文件:
Writing the array to the h5 file is straight forward:
In [231]: f = h5py.File('test1.h5', 'w')
In [232]: g = f.create_dataset('data', data=arr)
In [233]: g.dtype
Out[233]: dtype([('state', [('angle_rad', '<f4'), ('value2', '<f4')]), ('value1', '<f4'), ('value3', '<f4')])
In [234]: g[:]
Out[234]:
array([(( 0.034903, 0.83322001), 0.09, 0.30000001),
(( 1. , 1. ), 1. , 1. ),
(( 1. , 1. ), 1. , 1. )],
dtype=[('state', [('angle_rad', '<f4'), ('value2', '<f4')]), ('value1', '<f4'), ('value3', '<f4')])
从理论上讲,我们可以编写像write_layer
这样的函数来遍历您的字典并构造相关的dtype和记录.
In theory we could write functions like write_layer
that work through your dictionary and construct the relevant dtype and records.
这篇关于h5py-将对象动态写入文件吗?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!