确保我的程序没有进行并发文件写入 [英] Ensuring that my program is not doing a concurrent file write

问题描述

我正在编写一个脚本，该脚本需要对任何给定文件执行安全写入，即，如果已知没有其他进程正在写入文件，则附加一个文件.我对该理论的理解是，在文件系统上使用写锁可以防止并发写入，但在实践中似乎并非如此.

I am writing a script that is required to perform safe-writes to any given file i.e. append a file if no other process is known to be writing into it. My understanding of the theory was that concurrent writes were prevented using write locks on the file system but it seems not to be the case in practice.

以下是我设置测试用例的方式:我正在重定向 ping 命令的输出:

Here's how I set up my test case: I am redirecting the output of a ping command:

ping 127.0.0.1 > fileForSafeWrites.txt

另一方面，我有以下 python 代码尝试写入文件:

On the other end, I have the following python code attempting to write to the file:

handle = open('fileForSafeWrites.txt', 'w')
handle.write("Probing for opportunity to write")
handle.close()

同时运行两个进程优雅地完成.我看到 fileForSafeWrites.txt 已经变成了一个包含二进制内容的文件，而不是第一个进程发出的写锁，以防止它被 Python 代码写入.

Running concurrently both processes gracefully complete. I see that fileForSafeWrites.txt has turned into a file with binary content, instead of a write lock issued by the first process that protects it from being written into by the Python code.

如何强制我的一个或两个并发进程不相互干扰?我读过人们建议能够获取写入文件句柄作为文件被写入安全的证据，例如在 https://stackoverflow.com/a/3070749/1309045

How do I force either or both of my concurrent processes not to interfere with each other? I have read people advise the ability to get a write file handle as evidence for the file being write to safe, such as in https://stackoverflow.com/a/3070749/1309045

此行为是否特定于我的操作系统和 Python.我在 Ubuntu 12.04 环境下使用 Python2.7.

Is this behavior specific to my Operating System and Python. I use Python2.7 in an Ubuntu 12.04 environment.

推荐答案

灵感来自 为并发检查描述的解决方案，我想出了以下代码片段.如果能够适当地预测写入相关文件的频率，它就会起作用.解决方案是通过使用文件修改时间.

Inspired from a solution described for concurrency checks, I came up with the following snippet of code. It works if one is able to appropriately predict the frequency at which the file in question is written. The solution is through the use of file-modification times.

import os
import time

'''Find if a file was modified in the last x seconds given by writeFrequency.'''
def isFileBeingWrittenInto(filename, 
                       writeFrequency = 180, overheadTimePercentage = 20):

    overhead = 1+float(overheadTimePercentage)/100 # Add some buffer time
    maxWriteFrequency = writeFrequency * overhead
    modifiedTimeStart = os.stat(filename).st_mtime # Time file last modified
    time.sleep(writeFrequency)                     # wait writeFrequency # of secs
    modifiedTimeEnd = os.stat(filename).st_mtime   # File modification time again
    if 0 < (modifiedTimeEnd - modifiedTimeStart) <= maxWriteFrequency:
        return True
    else:
        return False

if not isFileBeingWrittenInto('fileForSafeWrites.txt'):
    handle = open('fileForSafeWrites.txt', 'a')
    handle.write("Text written safely when no one else is writing to the file")
    handle.close()

这不会进行真正的并发检查，但可以结合各种其他实用方法来安全地写入文件，而不必担心文本乱码.希望它可以帮助下一个寻找方法的人.

This does not do true concurrency checks but can be combined with a variety of other methods for practical purposes to safely write into a file without having to worry about garbled text. Hope it helps the next person searching for a way to do this.

编辑更新:

经过进一步测试，我遇到了一个高频写入过程，需要从修改条件逻辑

Upon further testing, I encountered a high-frequency write process that required the conditional logic to be modified from

if 0 < (modifiedTimeEnd - modifiedTimeStart) < maxWriteFrequency 

到

if 0 < (modifiedTimeEnd - modifiedTimeStart) <= maxWriteFrequency 

从理论上和实践上来说，这是一个更好的答案.

That makes a better answer, in theory and in practice.

这篇关于确保我的程序没有进行并发文件写入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！