在 Windows 上成功shutil.rmtree 后，os.mkdir 可能会因PermissionError 而失败 [英] os.mkdir can fail with PermissionError after successful shutil.rmtree on Windows

问题描述

考虑以下用于清理目录的 python 函数:

Consider the following python function for cleaning a directory:

def cleanDir(path):
  shutil.rmtree(path)
  os.mkdir(path)

在 Windows 上(实际在 Windows7 和 Windows10 上用 python 2.7.10 和 3.4.4 测试过)，当使用 Windows 资源管理器同时导航到相应目录时(或仅在左侧树窗格中导航到父文件夹时))，可能会引发以下异常:

On Windows (actually tested on Windows7 and Windows10 with python 2.7.10 and 3.4.4), when navigating at the same time with Windows Explorer into the corresponding directory (or when only navigating in the left tree pane to the parent folder), the following exception may be raised:

Traceback (most recent call last):
  ...
  File "cleanDir.py", line ..., in cleanDir
    os.mkdir(path)
PermissionError: [WinError 5] Access is denied: 'testFolder'

此问题已在此问题中报告.但是没有进一步分析，使用 sleep 的给定解决方案并不令人满意.根据 Eryk 在下面的评论，同样的行为也可以预期到当前的 python 版本，即 python 3.8.

The problem has already been reported in this issue. But it was not analyzed further and the given solution using sleep is not satisfying. According to Eryk's comments below the same behaviour is also to be expected up to current python versions, i.e. python 3.8.

请注意，shutil.rmtree 无一例外地返回.但是尝试立即再次创建目录可能会失败.(重试大多数情况下是成功的，请参阅下面的完整测试代码.)并注意，您需要在 Windows 资源管理器中左右单击测试文件夹的左侧和右侧，以强制出现问题.

Note that shutil.rmtree returns without exception. But trying to create the directory again immediately may fail. (A retry is most of the time successful, see the full code for testing below.) And note that you need to click around in the Windows Explorer in the test folders, left and right side, to force the problem.

问题似乎出在 Windows 文件系统 API 函数中(而不是在 Python os 模块中):当 Windows 资源管理器具有处理相应的文件夹.

The problem seems to be in the Windows filesystem API functions (and not in the Python os module): deleted folders seem not to be "forwarded" to all functions immediately, when Windows Explorer has a handle on the corresponding folder.

import os, shutil
import time

def populateFolder(path):
  if os.path.exists(path):
    with open(os.path.join(path,'somefile.txt'), 'w') as f:
      f.write('test')
  #subfolderpath = os.path.join(path,'subfolder')
  #os.mkdir(subfolderpath)
  #with open(os.path.join(subfolderpath,'anotherfile.txt'), 'w') as f2:
  #  f2.write('test')

def cleanDir(path):
  shutil.rmtree(path)
  os.mkdir(path)


def cleanDir_safe(path):
  shutil.rmtree(path)

  try:
    #time.sleep(0.005) # makes first try of os.mkdir successful
    os.mkdir(path)
  except Exception as e:
    print('os.mkdir failed: %s' % e)
    time.sleep(0.01)
    os.mkdir(path)

  assert os.path.exists(path)


FOLDER_PATH = 'testFolder'
if os.path.exists(FOLDER_PATH):
  cleanDir(FOLDER_PATH)
else:
  os.mkdir(FOLDER_PATH)

loopCnt = 0
while True:
  populateFolder(FOLDER_PATH)
  #cleanDir(FOLDER_PATH)
  cleanDir_safe(FOLDER_PATH)
  time.sleep(0.01)
  loopCnt += 1
  if loopCnt % 100 == 0:
    print(loopCnt)

推荐答案

Explorer 具有共享删除/重命名访问权限的目录的打开句柄.这允许 rmdir 成功，而通常打开不会共享删除/重命名访问，并且 rmdir 会因共享冲突而失败 (32).然而，即使 rmdir 成功，在资源管理器关闭其句柄之前，该目录实际上并没有被取消链接.它正在监视目录的更改，因此它会收到目录已被删除的通知，但即使它立即关闭其句柄，脚本的 os.mkdir 调用也存在竞争条件.

Explorer has an open handle to the directory that shares delete/rename access. This allows rmdir to succeed, whereas normally an open would not share delete/rename access, and rmdir would fail with a sharing violation (32). However, even though rmdir succeeds, the directory doesn't actually get unlinked until Explorer closes its handle. It's watching the directory for changes, so it gets notified that the directory has been deleted, but even if it closes its handle immediately, there's a race condition with the script's os.mkdir call.

您应该在循环中重试 os.mkdir，并增加超时.您还需要一个 shutil.rmtree 的 onerror 处理程序来处理尝试删除非空目录，因为它包含已删除"的文件或目录.

You should retry os.mkdir in a loop, with an increasing timeout. You also need an onerror handler for shutil.rmtree that handles trying to remove a directory that's not empty because it contains 'deleted' files or directories.

例如:

import os
import time
import errno
import shutil

def onerror(function, path, exc_info):
    # Handle ENOTEMPTY for rmdir
    if (function is os.rmdir
          and issubclass(exc_info[0], OSError)
          and exc_info[1].errno == errno.ENOTEMPTY):
        timeout = 0.001
        while timeout < 2:
            if not os.listdir(path):
                return os.rmdir(path)
            time.sleep(timeout)
            timeout *= 2
    raise

def clean_dir_safe(path):
    shutil.rmtree(path, onerror=onerror)
    # rmtree didn't fail, but path may still be linked if there is or was
    # a handle that shares delete access. Assume the owner of the handle
    # is watching for changes and will close it ASAP. So retry creating
    # the directory by using a loop with an increasing timeout.
    timeout = 0.001
    while True:
        try:
            return os.mkdir(path)
        except PermissionError as e:
            # Getting access denied (5) when trying to create a file or
            # directory means either the caller lacks access to the
            # parent directory or that a file or directory with that
            # name exists but is in the deleted state. Handle both cases
            # the same way. Otherwise, re-raise the exception for other
            # permission errors, such as a sharing violation (32).
            if e.winerror != 5 or timeout >= 2:
                raise
            time.sleep(timeout)
            timeout *= 2

<小时>

讨论

在一般情况下，这个问题是避免的"，因为现有的打开不共享删除/重命名访问.在这种情况下，尝试删除文件或目录会因共享冲突(winerror 32)而失败.例如，如果一个目录作为进程的工作目录打开，则它不共享删除/重命名访问权限.对于常规文件，大多数程序仅共享读取/执行和写入/追加访问权限.

In common cases, this problem is 'avoided' because existing opens do not share delete/rename access. In this case, trying to delete a file or directory fails with a sharing violation (winerror 32). For example, if a directory is open as the working directory of a process, it doesn't share delete/rename access. For regular files, most programs only share read/execute and write/append access.

临时文件通常使用删除/重命名访问共享打开，尤其是当它们使用删除/重命名访问打开时(例如，使用关闭时删除标志打开).这是仍然链接但无法访问的已删除"文件的最常见原因.另一种情况是打开一个目录以观察更改(例如，请参阅 ReadDirectoryChangesW).通常，此打开将共享删除/重命名访问权限，这就是此问题中资源管理器的情况.

Temporary files are often opened with delete/rename access sharing, especially if they're opened with delete/rename access (e.g. opened with the delete-on-close flag). This is the most common cause of 'deleted' files that are still linked but are inaccessible. Another case is opening a directory to watch for changes (e.g. see ReadDirectoryChangesW). Typically this open will share delete/rename access, which is the situation with Explorer in this question.

对于 Unix 开发人员来说，声明一个文件被删除而不被取消链接可能听起来很奇怪(至少可以这么说).在 Windows 中，删除文件(或目录)只是在其文件控制块 (FCB) 上设置删除配置.当文件系统清除文件的最后一个内核文件对象引用时，具有删除处置集的文件将自动取消链接.文件对象通常由 CreateFileW 创建，它返回对象的句柄.当文件对象的最后一个句柄关闭时，将触发文件对象的清理.由于子进程中的句柄继承或显式的 DuplicateHandle 调用，文件对象的多个句柄引用可能存在.

Stating that a file gets deleted without getting unlinked probably sounds strange (to say the least) to a Unix developer. In Windows, deleting a file (or directory) is just setting a delete disposition on its file control block (FCB). A file that has its delete disposition set gets automatically unlinked when the filesystem cleans up the file's last kernel file-object reference. A file object typically gets created by CreateFileW, which returns a handle to the object. Cleanup of a file object is triggered when the last handle to it is closed. Multiple handle references for a file object may exist due to handle inheritance in child processes or explicit DuplicateHandle calls.

重申一下，一个文件或目录可能被多个内核文件对象引用，每个内核文件对象可能被多个句柄引用.通常，使用经典的 Windows 删除语义，必须在取消链接文件之前关闭所有句柄.此外，设置删除处置不一定是最终的.如果任何打开的句柄具有删除/重命名访问权限，它实际上可以用于通过清除删除配置来恢复对文件的访问(例如，请参阅 SetFileInformationByHandle:FileDispositionInfo).

To reiterate, a file or directory may be referenced by multiple kernel file objects, for which each may be referenced by multiple handles. Normally, with classic Windows delete semantics, all of the handles have to be closed before the file gets unlinked. Moreover, setting the delete disposition isn't necessarily final. If any of the open handles has delete/rename access, it can actually be used to restore access to the file by clearing the delete disposition (e.g. see SetFileInformationByHandle: FileDispositionInfo).

在 Windows 10 中，内核还支持 POSIX 删除语义，一旦删除句柄关闭，文件或目录就会立即解除链接(请参阅 NTAPI FileDispositionInformationEx).NTFS 已更新以支持 POSIX 删除语义.最近 WINAPI DeleteFileW(即 Python os.remove)已切换到使用它，如果文件系统支持它，但 RemoveDirectoryW(即 Python os.rmdir) 仍然仅限于经典的 Windows 删除.

In Windows 10, the kernel also supports POSIX delete semantics, with which a file or directory is immediately unlinked as soon as the deleting handle is closed (see the details for NTAPI FileDispositionInformationEx). NTFS has been updated to support POSIX delete semantics. Recently WINAPI DeleteFileW (i.e. Python os.remove) has switched to using it if the filesystem supports it, but RemoveDirectoryW (i.e. Python os.rmdir) is still limited to a classic Windows delete.

实现 POSIX 语义对于 NTFS 来说相对容易.它只是设置删除配置并将文件重命名为 NTFS 保留目录\$Extend\$Deleted"，名称基于其文件 ID.在实践中，该文件似乎未链接，同时继续允许现有文件对象访问该文件.与经典删除相比，一个显着的区别是原始名称丢失了，因此具有删除/重命名访问权限的现有句柄无法取消删除处置.

Implementing POSIX semantics is relatively easy for NTFS. It simply sets the delete disposition and renames the file out of the way into an NTFS reserved directory "\$Extend\$Deleted", with a name that's based on its file ID. In practice, it appears that the file was unlinked, while continuing to allow existing file objects to access the file. One significant difference compared to a classic delete is that the original name is lost, so the delete disposition cannot be unset by existing handles that have delete/rename access.

这篇关于在 Windows 上成功shutil.rmtree 后，os.mkdir 可能会因PermissionError 而失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！