如何逐行读取大文件? [英] How to read a large file - line by line?
问题描述
我想遍历整个文件的每一行.一种方法是读取整个文件,将其保存到列表中,然后遍历感兴趣的行.这种方法占用大量内存,所以我正在寻找替代方法.
I want to iterate over each line of an entire file. One way to do this is by reading the entire file, saving it to a list, then going over the line of interest. This method uses a lot of memory, so I am looking for an alternative.
到目前为止我的代码:
for each_line in fileinput.input(input_file):
do_something(each_line)
for each_line_again in fileinput.input(input_file):
do_something(each_line_again)
执行此代码会给出错误消息:device active
.
Executing this code gives an error message: device active
.
有什么建议吗?
目的是计算成对的字符串相似度,这意味着对于文件中的每一行,我想计算与每隔一行的 Levenshtein 距离.
The purpose is to calculate pair-wise string similarity, meaning for each line in file, I want to calculate the Levenshtein distance with every other line.
推荐答案
正确的、完全 Pythonic 的文件读取方式如下:
The correct, fully Pythonic way to read a file is the following:
with open(...) as f:
for line in f:
# Do something with 'line'
with
语句处理打开和关闭文件,包括内部块中是否引发异常.for line in f
将文件对象 f
视为可迭代对象,它会自动使用缓冲 I/O 和内存管理,因此您不必担心大文件.
The with
statement handles opening and closing the file, including if an exception is raised in the inner block. The for line in f
treats the file object f
as an iterable, which automatically uses buffered I/O and memory management so you don't have to worry about large files.
应该有一种——最好只有一种——显而易见的方法.
There should be one -- and preferably only one -- obvious way to do it.
这篇关于如何逐行读取大文件?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!