连接大量 HDF5 文件 [英] Concatenate a large number of HDF5 files

问题描述

我有大约 500 个 HDF5 文件，每个文件大约 1.5 GB.

I have about 500 HDF5 files each of about 1.5 GB.

每个文件都具有相同的精确结构，即 7 个​​复合(int、double、double)数据集和可变数量的样本.

Each of the files has the same exact structure, which is 7 compound (int,double,double) datasets and variable number of samples.

现在我想通过连接每个数据集来连接所有这些文件，这样最后我就有一个包含 7 个数据集的 750 GB 文件.

Now I want to concatenate all this files by concatenating each of the datasets so that at the end I have a single 750 GB file with my 7 datasets.

目前我正在运行一个 h5py 脚本:

Currently I am running a h5py script which:

• 创建一个具有无限最大值的正确数据集的 HDF5 文件
• 依次打开所有文件
• 检查样本数量是多少(因为它是可变的)
• 调整全局文件的大小
• 附加数据

这显然需要好几个小时，你有什么改进的建议吗?

this obviously takes many hours, would you have a suggestion about improving this?

我正在开发一个集群，所以我可以并行使用 HDF5，但是我在 C 编程方面还不够好，无法自己实现某些东西，我需要一个已经编写好的工具.

I am working on a cluster, so I could use HDF5 in parallel, but I am not good enough in C programming to implement something myself, I would need a tool already written.

推荐答案

我发现大部分时间都花在调整文件大小上，因为我在每一步都在调整大小，所以我现在首先浏览所有文件并获取它们的长度(它是可变的).

I found that most of the time was spent in resizing the file, as I was resizing at each step, so I am now first going trough all my files and get their length (it is variable).

然后我创建全局 h5file，将总长度设置为所有文件的总和.

Then I create the global h5file setting the total length to the sum of all the files.

只有在这个阶段之后，我才用所有小文件中的数据填充 h5file.

Only after this phase I fill the h5file with the data from all the small files.

现在每个文件大约需要 10 秒，因此应该不到 2 小时，而之前需要更多时间.

now it takes about 10 seconds for each file, so it should take less than 2 hours, while before it was taking much more.

这篇关于连接大量 HDF5 文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！