创建HDF5文件而不关闭文件时损坏文件(h5py) [英] Corrupt files when creating HDF5 files without closing them (h5py)

问题描述

我正在使用h5py将实验数据存储在HDF5容器中.

I am using h5py to store experiment data in an HDF5 container.

在交互式会话中，我使用以下命令打开文件:

In an interactive session I open the file using:

measurement_data = h5py.File('example.hdf5', 'a')

然后，我使用一些自写函数将数据写入文件(经过几天的实验，可能会获得许多GB的数据).在实验结束时，我通常会使用

Then I write data to the file using some self-written functions (can be many GB of data from a couple of days experiment). At the end of the experiment I usually would close the file using

measurement_data.close()

不幸的是，交互会话有时会在我没有明确关闭文件的情况下结束(意外终止了该会话，断电，由于某些其他软件而导致的OS崩溃).这始终会导致文件损坏和完整数据丢失.当我尝试打开它时，出现错误:

Unfortunately, from time to time it happens, that the interactive session ends without me explicitly closing the file (accidentally killing the session, power outage, crash of OS due to some other software). This always results in a corrupt file and loss of the complete data. When I try to open it, I get the error:

OSError: Unable to open file (File signature not found)

我也无法在HDFview或我尝试过的任何其他软件中打开文件.

I also cannot open the file in HDFview, or any other software I tried.

1. 即使没有明确关闭文件，也可以避免文件损坏吗?我已经阅读了在此处的使用，但是我不确定在会话意外结束时是否有帮助.
2. 我可以通过某种方式恢复损坏文件中的数据吗?是否有维修程序?

对于我来说，每次打开和关闭文件总是很不利，因为我不断从许多不同的函数和线程中写入数据.因此，我会对其他解决方案感到满意.

Always opening and closing the file for every write access sounds pretty unfavorable to me, because I am continuously writing data from many different functions and threads. So I'd be more happy with a different solution.

推荐答案

HDF5设计人员知道损坏问题.他们正在通过添加日志记录来解决此问题.同时，您可以定期调用flush()以确保已清除所有写操作，这可以最大程度地减少部分损坏.您也可以尝试使用外部链接，您可以将数据片段存储在单独的文件中，但在读取数据时将它们链接为一个结构.

The corruption problem is known to the HDF5 designers. They are working on fixing this in version 1.10 by adding journalling. In the mean time you can call flush() periodically to make sure your writes have been flushed, which should minimise some of the damage. You can also try to use external links which will allow you to store pieces of data in separate files but link them together into one structure when you read them.

这篇关于创建HDF5文件而不关闭文件时损坏文件(h5py)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！