从/向二进制文件读取/写入Python列表 [英] Read/Write Python List from/to Binary file

问题描述

根据Python Cookbook，以下是将元组列表写入二进制文件的方法:

According to Python Cookbook, below is how to write a list of tuple into binary file:

from struct import Struct
def write_records(records, format, f):
    '''
    Write a sequence of tuples to a binary file of structures.
    '''
    record_struct = Struct(format)
    for r in records:
        f.write(record_struct.pack(*r))

# Example
if __name__ == '__main__':
    records = [ (1, 2.3, 4.5),
                (6, 7.8, 9.0),
                (12, 13.4, 56.7) ]
    with open('data.b', 'wb') as f:
        write_records(records, '<idd', f)

而且效果很好. 为了读取(大量二进制数据)，作者建议以下内容:

And it works well. For reading (large amount of binary data), the author recommended the following:

>>> import numpy as np
>>> f = open('data.b', 'rb')
>>> records = np.fromfile(f, dtype='<i,<d,<d')
>>> records
array([(1, 2.3, 4.5), (6, 7.8, 9.0), (12, 13.4, 56.7)],
dtype=[('f0', '<i4'), ('f1', '<f8'), ('f2', '<f8')])
>>> records[0]
(1, 2.3, 4.5)
>>> records[1]
(6, 7.8, 9.0)
>>>

这也很好，但是这个record不是普通的numpy数组.例如，type(record[0])将返回<type 'numpy.void'>.更糟糕的是，我无法使用X = record[:, 0]提取第一列.

It is also good, but this record is not a normal numpy array. For instance, type(record[0]) will return <type 'numpy.void'>. Even worse, I cannot extract the first column using X = record[:, 0].

是否有一种方法可以有效地将列表(或任何其他类型)从二进制文件加载到普通的numpy数组中? 提前谢谢.

Is there a way to efficiently load list(or any other types) from binary file into a normal numpy array? Thx in advance.

推荐答案

In [196]: rec = np.fromfile('data.b', dtype='<i,<d,<d')
In [198]: rec
Out[198]: 
array([( 1,   2.3,   4.5), ( 6,   7.8,   9. ), (12,  13.4,  56.7)], 
      dtype=[('f0', '<i4'), ('f1', '<f8'), ('f2', '<f8')])

这是一维结构化数组

In [199]: rec['f0']
Out[199]: array([ 1,  6, 12], dtype=int32)
In [200]: rec.shape
Out[200]: (3,)
In [201]: rec.dtype
Out[201]: dtype([('f0', '<i4'), ('f1', '<f8'), ('f2', '<f8')])

请注意，其tolist看起来与原始的records相同:

Note that its tolist looks identical to your original records:

In [202]: rec.tolist()
Out[202]: [(1, 2.3, 4.5), (6, 7.8, 9.0), (12, 13.4, 56.7)]
In [203]: records
Out[203]: [(1, 2.3, 4.5), (6, 7.8, 9.0), (12, 13.4, 56.7)]

您可以使用以下任一列表创建二维数组:

You could create a 2d array from either list with:

In [204]: arr2 = np.array(rec.tolist())
In [205]: arr2
Out[205]: 
array([[  1. ,   2.3,   4.5],
       [  6. ,   7.8,   9. ],
       [ 12. ,  13.4,  56.7]])
In [206]: arr2.shape
Out[206]: (3, 3)

还有其他方法可以将结构化数组转换为常规"数组，但这是最简单，最一致的.

There are other ways of converting a structured array to 'regular' array, but this is simplest and most consistent.

常规数组的tolist使用嵌套列表.结构化版本中的元组旨在传达差异:

The tolist of a regular array uses nested lists. The tuples in the structured version are intended to convey a difference:

In [207]: arr2.tolist()
Out[207]: [[1.0, 2.3, 4.5], [6.0, 7.8, 9.0], [12.0, 13.4, 56.7]]

在结构化数组中，第一个字段是整数.在常规数组中，第一列与其他列相同，即浮点数.

In the structured array the first field is integer. In the regular array the first column is same as the others, float.

如果二进制文件包含所有浮点数，则可以将其加载为1d的浮点数并进行整形

If the binary file contained all floats, you could load it as a 1d of floats and reshape

In [208]: with open('data.f', 'wb') as f:
     ...:         write_records(records, 'ddd', f)
In [210]: rec2 = np.fromfile('data.f', dtype='<d')
In [211]: rec2
Out[211]: array([  1. ,   2.3,   4.5,   6. ,   7.8,   9. ,  12. ,  13.4,  56.7])

但是要利用二进制文件中的任何记录结构，您也需要按记录加载，这意味着结构化数组:

But to take advantage of any record structure in the binary file, you have load by records as well, which means structured array:

In [213]: rec3 = np.fromfile('data.f', dtype='d,d,d')
In [214]: rec3
Out[214]: 
array([(  1.,   2.3,   4.5), (  6.,   7.8,   9. ), ( 12.,  13.4,  56.7)], 
      dtype=[('f0', '<f8'), ('f1', '<f8'), ('f2', '<f8')])

这篇关于从/向二进制文件读取/写入Python列表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！