使用h5py保存pandas DataFrame与其他HDF5读卡器的互操作性 [英] Save pandas DataFrame using h5py for interoperabilty with other hdf5 readers
问题描述
这是一个示例数据框:
import pandas as pd
NaN = float('nan')
ID = [1, 2, 3, 4, 5, 6, 7]
A = [NaN, NaN, NaN, 0.1, 0.1, 0.1, 0.1]
B = [0.2, NaN, 0.2, 0.2, 0.2, NaN, NaN]
C = [NaN, 0.5, 0.5, NaN, 0.5, 0.5, NaN]
columns = {'A':A, 'B':B, 'C':C}
df = pd.DataFrame(columns, index=ID)
df.index.name = 'ID'
print(df)
A B C
ID
1 NaN 0.2 NaN
2 NaN NaN 0.5
3 NaN 0.2 0.5
4 0.1 0.2 NaN
5 0.1 0.2 0.5
6 0.1 NaN 0.5
7 0.1 NaN NaN
我知道pandas具有基于pytables的HDFStore,这是有效序列化/反序列化数据帧的简便方法.但是,使用阅读器h5py或matlab直接加载这些数据集并不容易.如何使用h5py保存数据帧,以便可以使用其他hdf5阅读器轻松加载回该数据帧?
I know that pandas has the pytables based HDFStore, which is an easy way to efficiently serialize/deserialize a data frame. But those datasets are not very easy to load directly using a reader h5py or matlab. How can I save a data frame using h5py, so that I can easily load it back using another hdf5 reader?
推荐答案
pandas HDFStore
格式是标准的HDF5,只是有关如何解释元数据的约定.文档位于此处
The pandas HDFStore
format is standard HDF5, with just a convention for how to interpret the meta-data. Docs are here
In [54]: df.to_hdf('test.h5','df',mode='w',format='table',data_columns=True)
In [55]: h = h5py.File('test.h5')
In [56]: h['df']['table']
Out[56]: <HDF5 dataset "table": shape (7,), type "|V32">
In [64]: h['df']['table'][:]
Out[64]:
array([(1, nan, 0.2, nan), (2, nan, nan, 0.5), (3, nan, 0.2, 0.5),
(4, 0.1, 0.2, nan), (5, 0.1, 0.2, 0.5), (6, 0.1, nan, 0.5),
(7, 0.1, nan, nan)],
dtype=[('index', '<i8'), ('A', '<f8'), ('B', '<f8'), ('C', '<f8')])
In [57]: h['df']['table'].attrs.items()
Out[57]:
[(u'CLASS', 'TABLE'),
(u'VERSION', '2.7'),
(u'TITLE', ''),
(u'FIELD_0_NAME', 'index'),
(u'FIELD_1_NAME', 'A'),
(u'FIELD_2_NAME', 'B'),
(u'FIELD_3_NAME', 'C'),
(u'FIELD_0_FILL', 0),
(u'FIELD_1_FILL', 0.0),
(u'FIELD_2_FILL', 0.0),
(u'FIELD_3_FILL', 0.0),
(u'index_kind', 'integer'),
(u'A_kind', "(lp1\nS'A'\na."),
(u'A_meta', 'N.'),
(u'A_dtype', 'float64'),
(u'B_kind', "(lp1\nS'B'\na."),
(u'B_meta', 'N.'),
(u'B_dtype', 'float64'),
(u'C_kind', "(lp1\nS'C'\na."),
(u'C_meta', 'N.'),
(u'C_dtype', 'float64'),
(u'NROWS', 7)]
In [58]: h.close()
任何HDF5阅读器都可以完全读取数据.对某些元数据进行了腌制,因此必须小心.
The data will be completely readable in any HDF5 reader. Some of the meta-data is pickled, so care must be taken.
这篇关于使用h5py保存pandas DataFrame与其他HDF5读卡器的互操作性的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!