查找保存的numpy数组(.npy或.npz)的形状，而无需加载到内存中 [英] Finding shape of saved numpy array (.npy or .npz) without loading into memory

问题描述

我有一个巨大的压缩numpy数组保存到磁盘(内存中约20gb，压缩后更少).我需要知道此数组的形状，但是我没有可用的内存来加载它.如何在不将numpy数组加载到内存的情况下找到它的形状?

I have a huge compressed numpy array saved to disk (~20gb in memory, much less when compressed). I need to know the shape of this array, but I do not have the available memory to load it. How can I find the shape of the numpy array without loading it into memory?

推荐答案

在mmap_mode中打开文件可能会成功.

Opening the file in mmap_mode might do the trick.

    If not None, then memory-map the file, using the given mode
    (see `numpy.memmap` for a detailed description of the modes).
    A memory-mapped array is kept on disk. However, it can be accessed
    and sliced like any ndarray.  Memory mapping is especially useful for
    accessing small fragments of large files without reading the entire
    file into memory.

也可以在不读取数据缓冲区的情况下读取标头块，但这需要进一步挖掘底层的lib/npyio/format代码.我在最近关于将多个数组存储在单个文件中(并读取它们)的SO问题中对此进行了探讨.

It is also possible to read the header block without reading the data buffer, but that requires digging further into the underlying lib/npyio/format code. I explored that in a recent SO question about storing multiple arrays in a single file (and reading them).

https://stackoverflow.com/a/35752728/901925

这篇关于查找保存的numpy数组(.npy或.npz)的形状，而无需加载到内存中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！