Python:同时遍历许多大文件，获得第k行 [英] Python: Iterate over many large files simultaneusly, get every k-th line

问题描述

与标题中一样-我有很多非常大的文本文件(> 10GB)，它们具有相同的重复结构.我想过滤掉一些信息，所以我想从它们中产生第k行，但同时遍历它们.我尝试过itertools:islice和izip，但是我无法将它们放在一起...

As in the title - I have many very large text files (>10GB) that have the same, repetitive structure. I would like to filter some information out so I would like to yield every k-th line from them but iterating over them all at the same time. I have tried itertools: islice and izip, but I cannot put them together...

推荐答案

鉴于您在谈论使用itertools.izip()，我假设您在这里使用的是Python 2.

Given that you talk about using itertools.izip(), I'm going to assume you are using Python 2 here.

使用 itertools.islice() 可以方便地从文件，以及 itertools.izip_longest()函数来懒洋洋地结合阅读并处理较短的文件:

Use itertools.islice() to facilitate skipping lines from files, and the itertools.izip_longest() function to lazily combine reading in parallel as well as handle files that are shorter:

from itertools import islice, izip_longest

filenames = [fname1, fname2, fname3]
open_files = [open(fname) for fname in filenames]
kth_slice_files = (islice(f, None, None, k) for f in open_files)
try:
    for kth_lines in izip_longest(*kth_slice_files, fillvalue=''):
        # do something with those combined lines

islice(fileobj, None, None, k)将在 first 行开始，然后跳过k - 1行，给您1 + k，然后是1 + 2 * k等行.如果需要从下一行开始，请用该起始值替换第一个None.

islice(fileobj, None, None, k) will start at the first line, then skip k - 1 lines to give you the 1 + k, then 1 + 2*k, etc. lines. If you need to start at a later line, replace the first None with that starting value.

这篇关于Python:同时遍历许多大文件，获得第k行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！