线程安全和容错文件写入 [英] Threadsafe and fault-tolerant file writes

问题描述

我有一个长期运行的过程，该过程将很多东西写入文件中.结果应该是全部或什么都没有，因此我正在写入一个临时文件，并在最后将其重命名为真实名称.目前，我的代码是这样的:

I have a long-running process which writes a lot of stuff in a file. The result should be everything or nothing, so I'm writing to a temporary file and rename it to the real name at the end. Currently, my code is like this:

filename = 'whatever'
tmpname = 'whatever' + str(time.time())

with open(tmpname, 'wb') as fp:
    fp.write(stuff)
    fp.write(more stuff)

if os.path.exists(filename):
    os.unlink(filename)
os.rename(tmpname, filename)

我对此不满意，原因有几个:

I'm not happy with that for several reasons:

• 如果发生异常，则无法正确清除
• 它忽略并发问题
• 它不可重用(我在程序的不同位置需要它)

任何建议如何改善我的代码?有图书馆可以帮助我吗?

Any suggestions how to improve my code? Is there a library that can help me out?

推荐答案

您可以使用Python的tempfile模块为您提供一个临时文件名.它可以以线程安全的方式创建一个临时文件，而不是使用time.time()进行创建，如果同时在多个线程中使用该文件，则可能返回相同的名称.

You can use Python's tempfile module to give you a temporary file name. It can create a temporary file in a thread safe manner rather than making one up using time.time() which may return the same name if used in multiple threads at the same time.

正如对您的问题的评论中所建议的，这可以与上下文管理器一起使用.通过查看Python tempfile.py源，您可以了解一些有关如何实现想要执行的操作的想法.

As suggested in a comment to your question, this can be coupled with the use of a context manager. You can get some ideas of how to implement what you want to do by looking at Python tempfile.py sources.

以下代码段可能会做您想要的.它使用从tempfile返回的对象的某些内部.

The following code snippet may do what you want. It uses some of the internals of the objects returned from tempfile.

• 创建临时文件是线程安全的.
• 成功完成文件重命名是原子的，至少在Linux上是如此. os.path.exists()和os.rename()之间没有单独的检查，这可能会导致竞争状况.对于Linux上的原子重命名，源和目标必须位于同一文件系统上，这就是为什么此代码将临时文件与目标文件放置在同一目录中的原因.
• 对于RenamedTemporaryFile类，大多数情况下的行为应类似于NamedTemporaryFile，除非使用上下文管理器关闭该类时，文件会重命名.

• Creation of temporary files is thread safe.
• Renaming of files upon successful completion is atomic, at least on Linux. There isn't a separate check between os.path.exists() and the os.rename() which could introduce a race condition. For an atomic rename on Linux the source and destinations must be on the same file system which is why this code places the temporary file in the same directory as the destination file.
• The RenamedTemporaryFile class should behave like a NamedTemporaryFile for most purposes except when it is closed using the context manager, the file is renamed.

示例:

import tempfile
import os

class RenamedTemporaryFile(object):
    """
    A temporary file object which will be renamed to the specified
    path on exit.
    """
    def __init__(self, final_path, **kwargs):
        tmpfile_dir = kwargs.pop('dir', None)

        # Put temporary file in the same directory as the location for the
        # final file so that an atomic move into place can occur.

        if tmpfile_dir is None:
            tmpfile_dir = os.path.dirname(final_path)

        self.tmpfile = tempfile.NamedTemporaryFile(dir=tmpfile_dir, **kwargs)
        self.final_path = final_path

    def __getattr__(self, attr):
        """
        Delegate attribute access to the underlying temporary file object.
        """
        return getattr(self.tmpfile, attr)

    def __enter__(self):
        self.tmpfile.__enter__()
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type is None:
            self.tmpfile.delete = False
            result = self.tmpfile.__exit__(exc_type, exc_val, exc_tb)
            os.rename(self.tmpfile.name, self.final_path)
        else:
            result = self.tmpfile.__exit__(exc_type, exc_val, exc_tb)

        return result

然后您可以像这样使用它:

You can then use it like this:

with RenamedTemporaryFile('whatever') as f:
    f.write('stuff')

在写入过程中，内容进入一个临时文件，退出时该文件被重命名.这段代码可能需要进行一些调整，但是总体思路应该可以帮助您入门.

During writing, the contents go to a temporary file, on exit the file is renamed. This code will probably need some tweaks but the general idea should help you get started.

这篇关于线程安全和容错文件写入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！