h5py是否将整个文件读入内存? [英] Does h5py read the whole file into memory?

问题描述

h5py是否将整个文件读入内存?

Does h5py read the whole file into the memory?

如果是的话，如果我有一个非常大的文件怎么办?

If so, what if I have a very very big file?

如果没有，如果每次我想要一个数据时都从硬盘中获取数据，会不会很慢?我怎样才能更快?

If not, will it be quite slow if I take data from hard disk every time I want a single data? How can I make it faster?

推荐答案

h5py是否将整个文件读入内存?

Does h5py read the whole file into the memory?

不，不是.特别是，切片(dataset[50:100])允许您将数据集的一部分加载到内存中.有关详细信息，请参见 h5py文档.

No, it does not. In particular, slicing (dataset[50:100]) allows you to load fractions of a dataset into memory. For details, see the h5py docs.

如果不是，那么每次我想要一个数据时都从硬盘上取数据会很慢吗?

If not, will it be quite slow if I take data from hard disk every time I want a single data?

通常，hdf5非常快.但是从内存中读取显然比从磁盘中读取更快.由您决定将多少数据集读入内存(dataset[:]加载整个数据集).

In general, hdf5 is very fast. But reading from memory is obviously faster than reading from disk. It's your decision how much of a dataset is read into memory (dataset[:] loads the whole dataset).

如何使其更快?

How can I make it faster?

如果您希望优化性能，则应阅读部分关于分块和压缩.还有一本书详细解释了这些内容(免责声明:我不是作者).

If you care to optimize performance, you should read the sections about chunking and compression. There's also a book that explains these things in detail (disclaimer: I'm not the author).

这篇关于h5py是否将整个文件读入内存?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！