确保我的程序没有执行并发文件写入 [英] Ensuring that my program is not doing a concurrent file write

问题描述

我正在编写一个脚本来执行安全写入任何给定的文件，即追加一个文件，如果没有其他进程已知写入。我对这个理论的理解是，在文件系统上使用写入锁来阻止并发写入，但实际上似乎并非如此。

以下是我设置测试用例的方法：
我正在重定向ping命令的输出：

  ping 127.0.0.1> fileForSafeWrites.txt 
  

另一方面，我有下面的python代码试图写入文件：

  handle = open（'fileForSafeWrites.txt'，'w'）
 handle.write（ ）
 handle.close（）
  

同时并行运行这两个进程都能正常完成。我发现fileForSafeWrites.txt已经变成了一个二进制内容的文件，而不是由第一个进程发出的写入锁，以防止它被Python代码写入。

我如何强迫我的并发进程中的任何一个或两个不相互干扰？我读过人们建议能够获得写文件句柄作为文件写入安全的证据，例如在 https://stackoverflow.com/a/3070749/1309045

这是特定于我的操作系统和Python的行为。我在Ubuntu 12.04环境中使用了Python2.7。

解决方案

受到解决方案描述为并发检查，我想出了以下代码片段。如果能够恰当地预测写入的文件的频率，它就可以工作。解决方法是使用文件修改时间。

  import os 
 import time 
 
'''查找是否在writeFrequency给出的最后x秒内修改了一个文件'''
 def isFileBeingWrittenInto（filename，
 writeFrequency = 180，overheadTimePercentage = 20）：
 
 overhead = 1 + float（overheadTimePercentage）/ 100＃添加一些缓冲时间
 maxWriteFrequency = writeFrequency *开销
 modifiedTimeStart = os.stat（filename）.st_mtime＃上次修改时间文件
 time.sleep（writeFrequency）＃wait writeFrequency秒数
 modifiedTimeEnd = os.stat（filename）.st_mtime＃再次修改文件
 if 0< （modifiedTimeEnd  -  modifiedTimeStart）< = maxWriteFrequency：
返回True 
 else：
返回False 
 
如果不是isFileBeingWrittenInto（'fileForSafeWrites.txt'）：$ b $ （b）handle = open（'fileForSafeWrites.txt'，'a'）
 handle.write（没有其他人写入文件时安全写入的文本）
 handle.close（）
  

这不会执行真正的并发检查，但可以与其他各种方法结合使用，以便安全地写入一个文件，而不必担心乱码文本。希望它有助于下一个人搜索的方式来做到这一点。

$ b 编辑更新：

进一步测试后，我遇到了一个高频写入过程，要求将条件逻辑从

 修改if 0 < （modifiedTimeEnd  -  modifiedTimeStart）< maxWriteFrequency 
  

到

  if 0< （modifiedTimeEnd  -  modifiedTimeStart）<= maxWriteFrequency 
  

在理论上和实践中，这是一个更好的答案。

I am writing a script that is required to perform safe-writes to any given file i.e. append a file if no other process is known to be writing into it. My understanding of the theory was that concurrent writes were prevented using write locks on the file system but it seems not to be the case in practice.

Here's how I set up my test case: I am redirecting the output of a ping command:

ping 127.0.0.1 > fileForSafeWrites.txt

On the other end, I have the following python code attempting to write to the file:

handle = open('fileForSafeWrites.txt', 'w')
handle.write("Probing for opportunity to write")
handle.close()

Running concurrently both processes gracefully complete. I see that fileForSafeWrites.txt has turned into a file with binary content, instead of a write lock issued by the first process that protects it from being written into by the Python code.

How do I force either or both of my concurrent processes not to interfere with each other? I have read people advise the ability to get a write file handle as evidence for the file being write to safe, such as in https://stackoverflow.com/a/3070749/1309045

Is this behavior specific to my Operating System and Python. I use Python2.7 in an Ubuntu 12.04 environment.

解决方案

Inspired from a solution described for concurrency checks, I came up with the following snippet of code. It works if one is able to appropriately predict the frequency at which the file in question is written. The solution is through the use of file-modification times.

import os
import time

'''Find if a file was modified in the last x seconds given by writeFrequency.'''
def isFileBeingWrittenInto(filename, 
                       writeFrequency = 180, overheadTimePercentage = 20):

    overhead = 1+float(overheadTimePercentage)/100 # Add some buffer time
    maxWriteFrequency = writeFrequency * overhead
    modifiedTimeStart = os.stat(filename).st_mtime # Time file last modified
    time.sleep(writeFrequency)                     # wait writeFrequency # of secs
    modifiedTimeEnd = os.stat(filename).st_mtime   # File modification time again
    if 0 < (modifiedTimeEnd - modifiedTimeStart) <= maxWriteFrequency:
        return True
    else:
        return False

if not isFileBeingWrittenInto('fileForSafeWrites.txt'):
    handle = open('fileForSafeWrites.txt', 'a')
    handle.write("Text written safely when no one else is writing to the file")
    handle.close()

This does not do true concurrency checks but can be combined with a variety of other methods for practical purposes to safely write into a file without having to worry about garbled text. Hope it helps the next person searching for a way to do this.

EDIT UPDATE:

Upon further testing, I encountered a high-frequency write process that required the conditional logic to be modified from

if 0 < (modifiedTimeEnd - modifiedTimeStart) < maxWriteFrequency 

to

if 0 < (modifiedTimeEnd - modifiedTimeStart) <= maxWriteFrequency 

That makes a better answer, in theory and in practice.

这篇关于确保我的程序没有执行并发文件写入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！