在Python中创建竞争条件文件夹 [英] Race-condition creating folder in Python

问题描述

我有一个urllib2缓存模块，由于以下代码而偶尔崩溃：

I have a urllib2 caching module, which sporadically crashes because of the following code:

if not os.path.exists(self.cache_location):
    os.mkdir(self.cache_location)

问题是，到第二行执行时，该文件夹可能已存在，并且会出错：

The problem is, by the time the second line is being executed, the folder may exist, and will error:

  File ".../cache.py", line 103, in __init__
    os.mkdir(self.cache_location)
OSError: [Errno 17] File exists: '/tmp/examplecachedir/'

这是因为该脚本同时被启动了很多次，由第三方代码我无法控制。

This is because the script is simultaneously launched numerous times, by third-party code I have no control over.

代码（在尝试修复该代码之前错误）可以在在github上找到

The code (before I attempted to fix the bug) can be found here, on github

我不能使用 tempfile.mkstemp ，解决使用随机命名的目录（ tempfile.py此处的源

I can't use the tempfile.mkstemp, as it solves the race condition by using a randomly named directory (tempfile.py source here), which would defeat the purpose of the cache.

我不想简单地丢弃该错误，因为如果该错误会引发相同的错误Errno 17，文件夹名称作为文件存在（一个不同的错误），例如：

I don't want to simply discard the error, as the same error Errno 17 error is raised if the folder name exists as a file (a different error), for example:

$ touch blah
$ python
>>> import os
>>> os.mkdir("blah")
Traceback (most recent call last):
  File "", line 1, in 
OSError: [Errno 17] File exists: 'blah'
>>>

我无法使用 threading.RLock

因此，我尝试编写一个基于文件的简单锁（可以在此处找到该版本），但这有一个问题：它会向上一级创建锁文件，因此 /tmp/example.lock 表示 / tmp / example / ，如果您使用 / tmp，则会中断/ 作为缓存目录（因为它试图使 /tmp.lock ）。.

So, I tried writing a simple file-based lock (that version can be found here), but this has a problem: it creates the lockfile one level up, so /tmp/example.lock for /tmp/example/, which breaks if you use /tmp/ as a cache dir (as it tries to make /tmp.lock)..

简而言之，我需要缓存对磁盘的 urllib2 响应。为此，我需要以一种多进程安全的方式访问一个已知目录（如果需要，可以创建一个目录）。需要在OS X，Linux和Windows上运行。

In short, I need to cache urllib2 responses to disc. To do this, I need to access a known directory (creating it, if required), in a multiprocess safe way. It needs to work on OS X, Linux and Windows.

有何想法？我能想到的唯一替代解决方案是使用SQLite3存储而不是文件重写缓存模块。

Thoughts? The only alternative solution I can think of is to rewrite the cache module using SQLite3 storage, rather than files.

推荐答案

在Python 3中.x，您可以使用 os.makedirs（path，exist_ok = True），如果该目录存在，则不会引发任何异常。如果存在与所请求目录名称相同的文件（ path ），则会引发 FileExistsError：[Errno 17] 。

In Python 3.x, you can use os.makedirs(path, exist_ok=True), which will not raise any exception if such directory exists. It will raise FileExistsError: [Errno 17] if a file exists with the same name as the requested directory (path).

通过以下方式验证：

import os

parent = os.path.dirname(__file__)

target = os.path.join(parent, 'target')

os.makedirs(target, exist_ok=True)
os.makedirs(target, exist_ok=True)

os.rmdir(target)

with open(target, 'w'):
    pass

os.makedirs(target, exist_ok=True)

这篇关于在Python中创建竞争条件文件夹的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！