将numpy矩阵追加到没有numpy标头的二进制文件中 [英] Append numpy matrix to binary file without numpy header

问题描述

我不断接收新数据作为numpy矩阵，我需要将它们附加到现有文件中.该文件的结构和数据类型是固定的，因此我需要使用python进行转换.

I continously receive new data as numpy matrices which I need to append to an existing file. Structure and data type of that file are fixed, so I need python to do the conversion for me.

对于单个矩阵，这有效:

For a single matrix, this works:

myArr = np.reshape(np.arange(15), (3,5))
myArr.tofile('bin_file.dat')

但是假设我想继续为现有文件添加越来越多的数组，那么 numpy.tofile 将覆盖在文件中找到的所有内容，而不是追加.

But suppose I'd want to keep appending the existing file with more and more arrays, then numpy.tofile will overwrite any content it finds in the file, instead of appending.

我发现我也可以继续这样做:

I found that I could also keep doing this:

bfile = open('bin_file.dat', 'ab')
np.save(bfile, myArr)
bfile.close()

已成功附加到二进制文件.但是另一方面， numpy.save 不存储原始二进制数据，而是保存一个标头(我认为)，该标头使该文件无法被外部软件读取(我需要使用 float32的原始二进制文件).

which successfully appends to the binary file. But numpy.save on the other hand does not store the raw binary data, but also saves a header (I assume) which makes the file unreadable by foreign software (I need raw binary with float32).

使用 numpy.fromfile 读取文件的现有内容，附加我的数据并再次保存它不是一种选择，因为文件会变得很大，并且所有I/O都将花费很长时间进行处理.

Reading in the existing content of the file with numpy.fromfile, appending my data and saving it again is not an option, since the file becomes very large and all the I/O would take forever to process.

是否有类似 numpy.tofile 的 append 模式?我目前还缺少其他可能性吗?

Is there anything like an append mode for numpy.tofile? Are there other possibilities I am currently missing?

推荐答案

可以附加到 tofile 文件:

In [344]: with open('test1',mode='ba+') as f:
     ...:     np.arange(3).tofile(f)
     ...:     np.arange(5).tofile(f)
     ...:     
In [345]: np.fromfile('test1',dtype=int)
Out[345]: array([0, 1, 2, 0, 1, 2, 3, 4])

这将保存不包含形状或dtype信息的数组数据.因此，负载必须指定 dtype .而任何重塑都取决于您.

This saves the array data without shape or dtype information. So the load has to specify dtype. And any reshaping is up to you.

这篇关于将numpy矩阵追加到没有numpy标头的二进制文件中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！