保存numpy数组，以便无需加载即可使用 [英] Saving numpy array such that it is readily available without loading

问题描述

我有一个20GB的图像库，以高维numpy数组存储.该库使我可以使用这些图像，而不必每次都重新生成它们.现在我的问题是np.load("mylibrary")花费的时间与生成几个这些图像所花费的时间一样多.因此，我的问题是:是否有一种方法可以存储一个numpy数组，使其无需加载就可以轻松访问?

I have a 20GB library of images stored as a high-dimensional numpy array. This library allows me to these use images without having to generate them anew each time. Now my problem is that np.load("mylibrary") takes as much time as it would take to generate a couple of those images. Therefore my question is: Is there a way to store a numpy array such that it is readily accessible without having to load it?

我正在使用PyCharm

I am using PyCharm

推荐答案

我建议 h5py 这是一个HDF5二进制数据格式的Pythonic接口.

I would suggest h5py which is a Pythonic interface to the HDF5 binary data format.

它使您可以存储大量的数值数据，并轻松地从NumPy中操纵该数据.例如，您可以将磁盘上存储的多TB数据集切片，就像它们是真实的NumPy数组一样.数千个数据集可以存储在一个文件中，可以根据需要进行分类和标记.

It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want.

您还可以使用 PyTables'.这是另一个适用于python和numpy的HDF5接口

You can also use PyTables'. It is another HDF5 interface for python and numpy

PyTables是用于管理分层数据集的软件包，旨在高效，轻松地处理大量数据.您可以下载PyTables并免费使用.您可以在此处访问文档，一些使用示例和演示文稿.

PyTables is a package for managing hierarchical datasets and designed to efficiently and easily cope with extremely large amounts of data. You can download PyTables and use it for free. You can access documentation, some examples of use and presentations here.

numpy.memap是另一个选项.但是，它会比hdf5慢.另一个条件是阵列应限制为2.5G

numpy.memap is another option. It however would be slower than hdf5. Another condition is that a array should be limited to 2.5G

这篇关于保存numpy数组，以便无需加载即可使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！