将 numpy memmap 刷新到 npy 文件 [英] Flushing numpy memmap to npy file

问题描述

有没有一种方法可以将 numpy memmap 数组保存到 .npy 文件中?显然，有一种方法可以从 .npy 文件中加载这样的数组，如下所示

Is there a method to save a numpy memmap array into a .npy file? Apparently, there is a method to load such an array from a .npy file as follows

data = numpy.load("input.npy", mmap_mode='r')

但是刷新文件并不等同于以 .npy 格式存储它.

but flushing the file is not equivalent to storing it in a .npy format.

如果刷新是唯一的方法，那么有没有办法推断存储数组的形状?我更喜欢在另一个脚本中自动存储和检索(可能再次作为 memmap)的动态形状.

If flushing is the only way to go then is there a way to infer the shape of the stored array? I would prefer to have dynamic shape which is automatically stored and retrieved (possibly as memmap again) in another script.

我已经在各个地方搜索过，但没有找到任何结果.我现在存储到 .npy 的方法是

I have searched on various places about this but didn't find get any result. I way to store into .npy I do now is

numpy.save(output.filename, output.copy())

这违背了使用 memmap 但保留了形状的想法.

which defeats the idea of using memmap but preserves the shape.

注意:我知道 hdf5 和 h5py，但我想知道是否有一个纯粹的 numpy 解决方案.

NOTE: I know about hdf5 and h5py but I was wondering if there is a pure numpy solution to this.

推荐答案

有没有办法推断存储数组的形状?

is there a way to infer the shape of the stored array?

否.就 np.memmap 而言，该文件只是一个缓冲区 - 它存储数组的内容，但不存储维度、dtype 等.除非以某种方式包含在其中，否则无法推断该信息数组本身.如果您已经创建了一个由简单二进制文件支持的 np.memmap，那么您需要将其内容写入磁盘上的新 .npy 文件.

No. As far as np.memmap is concerned the file is just a buffer - it stores the contents of the array, but not the dimensions, dtype etc. There's no way to infer that information unless it's somehow contained within the array itself. If you've already created an np.memmap backed by a simple binary file then you would need to write its contents to a new .npy file on disk.

您可以通过使用 numpy.lib.format.open_memmap:

You could avoid generating a copy in memory by opening the new .npy file as another memory-mapped array using numpy.lib.format.open_memmap:

import numpy as np
from numpy.lib.format import open_memmap

# a 10GB memory-mapped array
x = np.memmap('/tmp/x.mm', mode='w+', dtype=np.ubyte, shape=(int(1E10),))

# create a memory-mapped .npy file with the same dimensions and dtype
y = open_memmap('/tmp/y.npy', mode='w+', dtype=x.dtype, shape=x.shape)

# copy the array contents
y[:] = x[:]

这篇关于将 numpy memmap 刷新到 npy 文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！