逐批读取文件中的多行 [英] Read multiple lines from a file batch by batch
问题描述
我想知道是否有一种方法可以逐批读取文件中的多行.例如:
I would like to know is there a method that can read multiple lines from a file batch by batch. For example:
with open(filename, 'rb') as f:
for n_lines in f:
process(n_lines)
在此函数中,我想做的是:对于每次迭代,将从文件中逐批读取下n行.
In this function, what I would like to do is: for every iteration, next n lines will be read from the file, batch by batch.
因为一个文件太大.我想做的是逐部分阅读.
Because one single file is too big. What I want to do is to read it part by part.
推荐答案
itertools.islice
和两个arg iter
可以完成此操作,但这有点可笑:
itertools.islice
and two arg iter
can be used to accomplish this, but it's a little funny:
from itertools import islice
n = 5 # Or whatever chunk size you want
with open(filename, 'rb') as f:
for n_lines in iter(lambda: tuple(islice(f, n)), ()):
process(n_lines)
这将使 islice
一次保持 n
行(使用 tuple
实际上强制读取整个块)直到 f
已用尽,此时它将停止.如果文件中的行数不是 n
的偶数倍,则最后一块将小于 n
行.如果希望所有行都是单个字符串,请将 for
循环更改为:
This will keep islice
ing off n
lines at a time (using tuple
to actually force the whole chunk to be read in) until the f
is exhausted, at which point it will stop. The final chunk will be less than n
lines if the number of lines in the file isn't an even multiple of n
. If you want all the lines to be a single string, change the for
loop to be:
# The b prefixes are ignored on 2.7, and necessary on 3.x since you opened
# the file in binary mode
for n_lines in iter(lambda: b''.join(islice(f, n)), b''):
另一种方法是为此目的使用 izip_longest
,这避免了 lambda
函数:
Another approach is to use izip_longest
for the purpose, which avoids lambda
functions:
from future_builtins import map # Only on Py2
from itertools import izip_longest # zip_longest on Py3
# gets tuples possibly padded with empty strings at end of file
for n_lines in izip_longest(*[f]*n, fillvalue=b''):
# Or to combine into a single string:
for n_lines in map(b''.join, izip_longest(*[f]*n, fillvalue=b'')):
这篇关于逐批读取文件中的多行的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!