如何逐行读取大文件? [英] How to read a large file - line by line?

问题描述

我想遍历整个文件的每一行.一种方法是读取整个文件，将其保存到列表中，然后遍历感兴趣的行.这种方法占用大量内存，所以我正在寻找替代方法.

I want to iterate over each line of an entire file. One way to do this is by reading the entire file, saving it to a list, then going over the line of interest. This method uses a lot of memory, so I am looking for an alternative.

到目前为止我的代码:

for each_line in fileinput.input(input_file):
    do_something(each_line)

    for each_line_again in fileinput.input(input_file):
        do_something(each_line_again)

执行此代码会给出错误消息:device active.

Executing this code gives an error message: device active.

有什么建议吗?

目的是计算成对的字符串相似度，这意味着对于文件中的每一行，我想计算与每隔一行的 Levenshtein 距离.

The purpose is to calculate pair-wise string similarity, meaning for each line in file, I want to calculate the Levenshtein distance with every other line.

推荐答案

正确的、完全 Pythonic 的文件读取方式如下:

The correct, fully Pythonic way to read a file is the following:

with open(...) as f:
    for line in f:
        # Do something with 'line'

with 语句处理打开和关闭文件，包括内部块中是否引发异常.for line in f 将文件对象 f 视为可迭代对象，它会自动使用缓冲 I/O 和内存管理，因此您不必担心大文件.

The with statement handles opening and closing the file, including if an exception is raised in the inner block. The for line in f treats the file object f as an iterable, which automatically uses buffered I/O and memory management so you don't have to worry about large files.

应该有一种——最好只有一种——显而易见的方法.

There should be one -- and preferably only one -- obvious way to do it.

这篇关于如何逐行读取大文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！