将numpy.arrays增量添加到保存文件 [英] Incremently appending numpy.arrays to a save file

问题描述

我已经尝试了Hpaulji概述的这种方法，但是它似乎不起作用:

I've tried this method outlined by Hpaulji but it doesn't seem to working:

如何附加多个numpy文件在python中变成一个numpy文件

基本上，我要遍历一个生成器，对数组进行一些更改，然后尝试保存每个迭代的数组.

Basically, I'm iterating through a generator, making some changes to an array, and then trying to save the each iteration's array.

这是我的示例代码:

filename = 'testing.npy'

with open(filename, 'wb') as f:
    for x, _ in train_generator:
        prediction = base_model.predict(x)
        print(prediction[0,0,0,0:5])
        np.save(filename, prediction)

        current_iteration += 1
    if current_iteration == 5:
        break

在这里，我要进行5次迭代，所以我希望保存 5 个不同的数组.

Here, I'm going through 5 iterations, so I was hoping to save 5 different arrays.

出于调试目的，我打印了每个数组的一部分:

I printed out a portion of each array, for debugging purposes:

[ 0.  0.  0.  0.  0.]
[ 0.          3.37349415  0.          0.          1.62561738]
[  0.          20.28489304   0.           0.           0.        ]
[ 0.  0.  0.  0.  0.]
[  0.          21.98013496   0.           0.           0.        ]

但是，当我尝试加载数组时，如此处所述多次， 如何将多个numpy文件附加到一个python中的numpy文件，我收到了EOFERROR:

But when I tried to load the array, multiple times as noted here, How to append many numpy files into one numpy file in python, I'm getting an EOFERROR:

file = r'testing.npy'

with open(file,'rb') as f:
    arr = np.load(f)
    print(arr[0,0,0,0:5])
    arr = np.load(f)
    print(arr[0,0,0,0:5])

仅输出最后一个数组，然后输出EOFERROR:

It's only outputting the last array and then an EOFERROR:

[  0.          21.98013496   0.           0.           0.        ]
EOFError: Ran out of input

print(arr[0,0,0,0:5])

我原本希望保存所有5个数组，但是当我多次加载save .npy文件时，我只会得到最后一个数组.

I was expection all 5 arrays to be saved, but when I load the save .npy file multiple times, I only get the last array.

那么，我应该如何保存保存并将新数组附加到文件中?

So, how should I be saving saving and appending new array to a file?

使用'.npz'测试只会保存最后一个数组

Testing with '.npz' only saves last array

filename = 'testing.npz'

current_iteration = 0
with open(filename, 'wb') as f:
    for x, _ in train_generator:
        prediction = base_model.predict(x)
        print(prediction[0,0,0,0:5])
        np.savez(f, prediction)



        current_iteration += 1
        if current_iteration == 5:
            break


#loading

    file = 'testing.npz'

    with open(file,'rb') as f:
        arr = np.load(f)
        print(arr.keys())


>>>['arr_0']

推荐答案

所有对np.save的调用都使用文件名，而不是文件句柄. 由于不重复使用文件句柄，因此每个保存都将覆盖文件，而不是将数组附加到文件句柄.

All your calls to np.save use the filename, not the filehandle. Since you do not reuse the filehandle, each save overwrites the file instead of appending the array to it.

这应该有效:

filename = 'testing.npy'

with open(filename, 'wb') as f:
    for x, _ in train_generator:
        prediction = base_model.predict(x)
        print(prediction[0,0,0,0:5])
        np.save(f, prediction)

        current_iteration += 1
    if current_iteration == 5:
        break

虽然将多个数组存储在一个.npy文件中可能会有好处(我想在内存有限的情况下有好处)，但它们是

And while there may be advantages to storing multiple arrays in one .npy file (I imagine advantages in situations where memory is limited), they are technically meant to store one single array, and you can use .npz files (np.savez or np.savez_compressed) to store multiple arrays:

filename = 'testing.npz'
predictions = []
for (x, _), index in zip(train_generator, range(5)):
    prediction = base_model.predict(x)
    predictions.append(prediction)
np.savez(filename, predictions) # will name it arr_0
# np.savez(filename, predictions=predictions) # would name it predictions
# np.savez(filename, *predictions) # would name it arr_0, arr_1, …, arr_4

这篇关于将numpy.arrays增量添加到保存文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！