从文件中读取结构数组 [英] Readng an array of structures from file

问题描述

我有下一个任务:我需要从文件中读取一个结构数组.读取一个结构没有问题:

I have the next task: I need to read an array of structures from file. There is no problem to read one structure:

structFmt = "=64s 2L 3d"    # char[ 64 ] long[ 2 ] double [ 3 ]
structLen = struct.calcsize( structFmt )
f = open( "path/to/file", "rb" )
structBytes = f.read( structLen )
s = struct.unpack( structFmt, structBytes )

读取简单"类型的数组也没有问题:

Also there is no problem to read an array of "simple" types:

f = open( "path/to/file", "rb" )
a = array.array( 'i' )
a.fromfile( f, 1024 )

但是从文件中读取 1024 个结构 structFmt 存在问题(当然对我来说).我认为，读取 1024 次结构并将其附加到列表中是一种开销.我不想使用像 numpy 这样的外部依赖项.

But there is a problem (for me, of course) to read 1024 structures structFmt from file. I think, that it is an overhead to read 1024 times struct and append it to a list. I do not want to use external dependencies like numpy.

推荐答案

我会考虑对文件进行映射，然后使用 ctypes 类方法 from_buffer() 调用.这将映射 ctypes 定义的结构数组 http://docs.python.org/library/ctypes#ctypes-arrays.

I would look at mmaping the file and then using ctypes class method from_buffer() call. This will map the ctypes defined array of structs http://docs.python.org/library/ctypes#ctypes-arrays.

这将结构映射到 mmap 文件，而无需显式读取/转换和复制内容.

This maps the structs over the mmap file without having to explicitly read/convert and copy things.

我不知道最终结果是否合适.

I don't know if the end result will be appropriate though.

这里只是为了好玩，这是一个使用 mmap 的快速示例.(我使用 dd dd if=/dev/zero of=./test.dat bs=96 count=10240

Just for fun here is a quick example using mmap. (I created a file using dd dd if=/dev/zero of=./test.dat bs=96 count=10240

from ctypes import Structure
from ctypes import c_char, c_long, c_double
import mmap
import timeit


class StructFMT(Structure):
     _fields_ = [('ch',c_char * 64),('lo',c_long *2),('db',c_double * 3)]

d_array = StructFMT * 1024

def doit():
    f = open('test.dat','r+b')
    m = mmap.mmap(f.fileno(),0)
    data = d_array.from_buffer(m)

    for i in data:
        i.ch, i.lo[0]*10 ,i.db[2]*1.0   # just access each row and bit of the struct and do something, with the data.

    m.close()
    f.close()

if __name__ == '__main__':
    from timeit import Timer
    t = Timer("doit()", "from __main__ import doit")
    print t.timeit(number=10)

这篇关于从文件中读取结构数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！