Python如何一次读取N行 [英] Python how to read N number of lines at a time

问题描述

我正在编写代码，一次获取一个N行的巨大文本文件(几GB)，处理该批处理，然后移至下N行，直到完成整个文件. (我不在乎最后一批不是理想的尺寸).

I am writing a code to take an enormous textfile (several GB) N lines at a time, process that batch, and move onto the next N lines until I have completed the entire file. (I don't care if the last batch isn't the perfect size).

我一直在阅读有关使用itertools islice进行此操作的信息.我想我已经中途了:

I have been reading about using itertools islice for this operation. I think I am halfway there:

from itertools import islice
N = 16
infile = open("my_very_large_text_file", "r")
lines_gen = islice(infile, N)

for lines in lines_gen:
     ...process my lines...

麻烦的是我想处理下一批的16行，但是我遗漏了一些东西

The trouble is that I would like to process the next batch of 16 lines, but I am missing something

推荐答案

islice()可用于获取迭代器的下一个n项.因此，list(islice(f, n))将返回文件f的下一个n行的列表.在循环中使用它会以n行的块的形式为您提供文件.在文件末尾，列表可能会更短，最后调用将返回一个空列表.

islice() can be used to get the next n items of an iterator. Thus, list(islice(f, n)) will return a list of the next n lines of the file f. Using this inside a loop will give you the file in chunks of n lines. At the end of the file, the list might be shorter, and finally the call will return an empty list.

from itertools import islice
with open(...) as f:
    while True:
        next_n_lines = list(islice(f, n))
        if not next_n_lines:
            break
        # process next_n_lines

一种替代方法是使用石斑鱼模式:

with open(...) as f:
    for next_n_lines in izip_longest(*[f] * n):
        # process next_n_lines

这篇关于Python如何一次读取N行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！