逐批读取文件中的多行 [英] Read multiple lines from a file batch by batch

问题描述

我想知道是否有一种方法可以逐批读取文件中的多行.例如:

I would like to know is there a method that can read multiple lines from a file batch by batch. For example:

with open(filename, 'rb') as f:
    for n_lines in f:
        process(n_lines)

在此函数中，我想做的是:对于每次迭代，将从文件中逐批读取下n行.

In this function, what I would like to do is: for every iteration, next n lines will be read from the file, batch by batch.

因为一个文件太大.我想做的是逐部分阅读.

Because one single file is too big. What I want to do is to read it part by part.

推荐答案

itertools.islice 和两个arg iter 可以完成此操作，但这有点可笑:

itertools.islice and two arg iter can be used to accomplish this, but it's a little funny:

from itertools import islice

n = 5  # Or whatever chunk size you want
with open(filename, 'rb') as f:
    for n_lines in iter(lambda: tuple(islice(f, n)), ()):
        process(n_lines)

这将使 islice 一次保持 n 行(使用 tuple 实际上强制读取整个块)直到 f 已用尽，此时它将停止.如果文件中的行数不是 n 的偶数倍，则最后一块将小于 n 行.如果希望所有行都是单个字符串，请将 for 循环更改为:

This will keep isliceing off n lines at a time (using tuple to actually force the whole chunk to be read in) until the f is exhausted, at which point it will stop. The final chunk will be less than n lines if the number of lines in the file isn't an even multiple of n. If you want all the lines to be a single string, change the for loop to be:

    # The b prefixes are ignored on 2.7, and necessary on 3.x since you opened
    # the file in binary mode
    for n_lines in iter(lambda: b''.join(islice(f, n)), b''):

另一种方法是为此目的使用 izip_longest ，这避免了 lambda 函数:

Another approach is to use izip_longest for the purpose, which avoids lambda functions:

from future_builtins import map  # Only on Py2
from itertools import izip_longest  # zip_longest on Py3

    # gets tuples possibly padded with empty strings at end of file
    for n_lines in izip_longest(*[f]*n, fillvalue=b''):

    # Or to combine into a single string:
    for n_lines in map(b''.join, izip_longest(*[f]*n, fillvalue=b'')):

这篇关于逐批读取文件中的多行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！