如何有效地读取二进制文件？ [英] How to efficiently read binary files?

问题描述

我想在Python中处理大型二进制文件（> 2GB）。我在纯Python中使用原型玩了一下

并对代码进行了分析。

的大部分时间似乎是使用struct模块来回转换来回字符串

。有没有办法在Python中直接读入数组

的整数？


TIA


David Lees

解决方案

2006-04-30，David Lees< de ********* @ verizon.net>写道：

我想在Python中处理大型二进制文件（> 2GB）。我用纯Python玩原型并编写代码。大多数时间似乎都花费了使用struct模块来回转换字符串
。有没有办法在Python中直接读入整数数组？

也许numarray模块？


-

Grant Edwards
[e-mail]

Grant Edwards写道：

2006-04-30，David Lees< de ********* @ verizon。净>写道：

我想在Python中处理大型二进制文件（> 2GB）。我用纯Python玩原型并编写代码。大多数时间似乎都花费了使用struct模块来回转换字符串
。有没有办法直接读入Python中的整数数组？

也许numarray模块？

$ b请给$ b numpy代码换新代码。特别是，即使在64位平台上，numarray仍受限于32位API

，因此即使使用mmap

，也很难处理大于2GB的文件。 numpy在64位平台上删除了这个限制。 32位用户

仍然需要将文件拆分为< 2GB块。

http://numeric.scipy.org/
https://lists.sourceforge.net/lists/...mpy-discussion


- -

Robert Kern
ro ********* @ gmail .com


我开始相信整个世界都是一个谜，一个无害的谜团

他们疯狂地试图解释它，好像它有一个潜在的真相。

- Umberto Eco






/>

David Lees< de ********* @ verizon.net>写道：

我想在Python中处理大型二进制文件（> 2GB）。我用纯Python玩原型并编写代码。大多数时间似乎都花费了使用struct模块来回转换字符串
。有没有办法直接读入Python中的整数数组？

import array
x = array.array（''l''）
help（x.fromfile）

来自file的内置函数的帮助：


fromfile（...）

fromfile（f，n）

>
从文件对象f中读取n个对象，并将它们附加到数组的末尾

。也称为阅读。

Alex

I want to process large binary files (>2GB) in Python. I have played
around with prototypes in pure Python and profiled the code. Most of
the time seems to be spent converting back and forth to and from strings
using the struct module. Is there a way to directly read into an array
of integers in Python?

TIA

David Lees

解决方案

On 2006-04-30, David Lees <de*********@verizon.net> wrote:

I want to process large binary files (>2GB) in Python. I have played
around with prototypes in pure Python and profiled the code. Most of
the time seems to be spent converting back and forth to and from strings
using the struct module. Is there a way to directly read into an array
of integers in Python?

Perhaps the numarray module?

--
Grant Edwards
gr****@visi.com

Grant Edwards wrote:

On 2006-04-30, David Lees <de*********@verizon.net> wrote:

I want to process large binary files (>2GB) in Python. I have played
around with prototypes in pure Python and profiled the code. Most of
the time seems to be spent converting back and forth to and from strings
using the struct module. Is there a way to directly read into an array
of integers in Python?

Perhaps the numarray module?

numpy for new code, please. In particular, numarray is limited by 32-bit APIs
even on 64-bit platforms, so a >2GB file will be difficult to process even when
using mmap. numpy removed this restriction on 64-bit platforms. 32-bit users
will still have to split up the file into <2GB chunks, though.

http://numeric.scipy.org/
https://lists.sourceforge.net/lists/...mpy-discussion

--
Robert Kern
ro*********@gmail.com

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

David Lees <de*********@verizon.net> wrote:

I want to process large binary files (>2GB) in Python. I have played
around with prototypes in pure Python and profiled the code. Most of
the time seems to be spent converting back and forth to and from strings
using the struct module. Is there a way to directly read into an array
of integers in Python?

import array
x=array.array(''l'')
help(x.fromfile)

Help on built-in function fromfile:

fromfile(...)
fromfile(f, n)

Read n objects from the file object f and append them to the end of
the array. Also called as read.
Alex

这篇关于如何有效地读取二进制文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！