无法将DataFrame保存到HDF5(“对象标头消息太大") [英] Unable to save DataFrame to HDF5 ("object header message is too large")

问题描述

我在Pandas中有一个DataFrame:

I have a DataFrame in Pandas:

In [7]: my_df
Out[7]: 
<class 'pandas.core.frame.DataFrame'>
Int64Index: 34 entries, 0 to 0
Columns: 2661 entries, airplane to zoo
dtypes: float64(2659), object(2)

当我尝试将其保存到磁盘时:

When I try to save this to disk:

store = pd.HDFStore(p_full_h5)
store.append('my_df', my_df)

我得到:

  File "H5A.c", line 254, in H5Acreate2
    unable to create attribute
  File "H5A.c", line 503, in H5A_create
    unable to create attribute in object header
  File "H5Oattribute.c", line 347, in H5O_attr_create
    unable to create new attribute in header
  File "H5Omessage.c", line 224, in H5O_msg_append_real
    unable to create new message
  File "H5Omessage.c", line 1945, in H5O_msg_alloc
    unable to allocate space for message
  File "H5Oalloc.c", line 1142, in H5O_alloc
    object header message is too large

End of HDF5 error back trace

Can't set attribute 'non_index_axes' in node:
 /my_df(Group) u''.

为什么?

注意:如果很重要，DataFrame列名称是简单的小字符串:

Note: In case it matters, the DataFrame column names are simple small strings:

In[12]: max([len(x) for x in list(my_df.columns)])
Out{12]: 47

Pandas 0.11以及IPython，Python和HDF5的最新稳定版本已全部包含.

This is all with Pandas 0.11 and the latest stable version of IPython, Python and HDF5.

推荐答案

对于列的所有元数据，HDF5的标头限制为64kb.这包括名称，类型等.当您大约浏览2000列时，将用光空间来存储所有元数据.这是pytables的基本限制.我认为他们不会在短期内采取任何变通办法.您将不得不拆分表或选择其他存储格式.

HDF5 has a header limit of 64kb for all metadata of the columns. This include name, types, etc. When you go about roughly 2000 columns, you will run out of space to store all the metadata. This is a fundamental limitation of pytables. I don't think they will make workarounds on their side any time soon. You will either have to split the table up or choose another storage format.

这篇关于无法将DataFrame保存到HDF5(“对象标头消息太大")的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！