使用read()打开文件时出现内存错误 [英] Memory error when opening the file with read()

问题描述

我是python的新手，正在编辑一个需要打开文件的程序，但该文件超过1.5 Gb，因此出现内存错误. 代码是:

I'm new to python and I'm editing a program where I need to open I file but it's more than 1.5 Gb so I get memory error. Code is:

f=open('thumbdata3.dat','rb')
tdata = f.read()
f.close()

ss = '\xff\xd8'
se = '\xff\xd9'

count = 0
start = 0
while True:
    x1 = tdata.find(ss,start)
    if x1 < 0:
        break
    x2 = tdata.find(se,x1)
    jpg = tdata[x1:x2+1]
    count += 1
    fname = 'extracted%d03.jpg' % (count)
    fw = open(fname,'wb')
    fw.write(jpg)
    fw.close()
    start = x2+2

所以我得到了

MemoryError

MemoryError

在

tdata = f.read()

部分.如何修改读取时拆分文件的功能?

section. How do I modify a function to split a file while being read?

推荐答案

从描述中看来，内存占用是这里的问题.因此，我们可以使用生成器来减少数据的内存占用量，以便它一步一步地加载正在使用的数据部分.

From the description it seems that the memory footprint is the problem here. So we can use the generators to reduce the memory footprint of the data , so that it loads the part of data being used one by one.

from itertools import chain, islice

def piecewise(iterable, n):
    "piecewise(Python,2) => Py th on"
    iterable = iter(iterable)
    while True:
        yield chain([next(iterable)], islice(iterable, n-1))

l = ...
file_large = 'large_file.txt'
with open(file_large) as bigfile:
   for i, lines in enumerate(piecewise(bigfile, l)):
      file_split = '{}.{}'.format(file_large, i)
      with open(file_split, 'w') as f:
         f.writelines(lines)

这篇关于使用read()打开文件时出现内存错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！