Python在& gt; 20GB文本文件中替换一行 [英] Python Replace one line in >20GB text file

问题描述

我完全知道有很多解决此问题的方法.

I am fully aware that there were many approaches to this problem.

我需要的是一个简单的Python脚本，它将只替换大文本文件中的1行.

What I need is a simple Python script that would replace only 1 line in a large text file.

始终是从头开始的第四行.

It is always the fourth line from the beginning.

由于文件(实际上是文件)大于20GB，因此我不想将其加载到内存或创建副本，只需高效地替换一行即可.

As the file (actually, files) is bigger than 20GB, I don't want to load it to memory or create a copy, just replace one line efficiently.

在这方面的任何帮助，我都会很高兴.

I'll be glad for any help in this regard.

A.

PS.我知道vi可以做到，但是我需要它作为脚本，这样不兼容vi的人也可以做到这一点.

PS. I know vi can do it, but I need it as a script, so that someone non-vi-compatible would be able to do it as well.

推荐答案

您可以打开文件进行更新，也可以使用 mmap 作为其他建议的答案.有关如何在文件中间进行编辑的示例:

You can open a file for updating, or use mmap as the other answer suggested. Example on how to edit in the middle of a file:

def example(fname):
    f = open(fname, 'r+b')
    f.seek(100)
    f.write('foobar')
    f.close()

这将在文件的位置100的"foobar"中进行编辑.但是，在通常情况下，您编辑的行变得越来越长，您仍然必须从头到尾遍历整个文件(您只能在末尾而不是在头处扩展和截断文件).在这方面，Vi并不是魔术，同样的规则也适用于它.

That will edit in "foobar" at location 100 in the file. However in the general case where the line you edit becomes either longer and shorter, you still will have to go through the whole file all the way to the end (you can only extend and truncate a file at the end, not at the head). Vi is not magic in this regard, the same rules apply to it.

为简单起见，我将遍历整个文件并输出一个新的，已编辑的文件.您绝对不希望一次将所有内容都读到内存中.逐行执行直到需要编辑的行，然后逐行进行.

To keep it simple, I would iterate through the whole file and output a new, edited file. You definitely don't want to read it all into memory at once. Do it line by line until the line you need to edit, and block by block after that.

您还可以使用 ed 或 sed 命令，因为这些命令的脚本编写可能比 vi 更简单.

You can also use the ed or sed commands as these are arguably simpler to script than vi.

这篇关于Python在& gt; 20GB文本文件中替换一行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！