“unlink()"在 Windows 上的 Python 共享内存中不起作用 [英] 'unlink()' does not work in Python's shared_memory on Windows

问题描述

我正在使用 Python 3.8 的新 shared_memory 模块并且无法在不终止使用它的进程的情况下释放共享内存.

I am using Python 3.8's new shared_memory module and fail to free the shared memory without terminating the processes using it.

在创建和使用共享内存块shm后，我在所有进程中通过shm.close()关闭它，最后通过shm释放它.在主进程中取消链接.但是，资源监视器显示内存在程序终止之前不会释放.这对我来说是一个严重的问题，因为我的程序需要运行很长时间.可以使用以下程序在 Windows/Python 3.8 上重现该问题:

After creating and using a block shm of shared memory, I close it via shm.close() in all processes and finally free it via shm.unlink in the main process. However, the reseource monitor shows me that the memory is not freed up until the program is terminated. This is a serious problem for me, because my program needs to run for a long time. The problem can be reproduced on Windows/Python 3.8 with the following program:

from multiprocessing import shared_memory, Pool
from itertools import repeat
from time import sleep

def fun(dummy, name):
    
    # access shared memory
    shm = shared_memory.SharedMemory(name=name)
    
    # do work
    sleep(1)
    
    # release shared memory
    shm.close()
    
    return dummy

def meta_fun(pool):
    
    # create shared array
    arr = shared_memory.SharedMemory(create=True, size=500000000)
    
    # compute result
    result = sum(pool.starmap(fun, zip(range(10), repeat(arr.name))))
    
    # release and free memory
    arr.close()
    arr.unlink()
    
    return result

if __name__ == '__main__':
    
    # use one Pool for many method calls to save the time for repeatedly
    # creating processes
    with Pool() as pool:
        for i in range(100):
            print(meta_fun(pool))

注意:在执行此脚本时，您可能会很快填满整个记忆！观看虚拟内存"资源监视器中的面板.

Caution: when executing this script, you may quickly fill your entire memory! Watch the "virtual memory" panel in the resource monitor.

经过一番研究，我发现 (1) unlink() 函数 在 Windows 上什么都不做:

After doing some research, I found out that (1) the unlink() function does nothing on Windows:

def unlink(self):
    """Requests that the underlying shared memory block be destroyed.
    In order to ensure proper cleanup of resources, unlink should be
    called once (and only once) across all processes which have access
    to the shared memory block."""
    if _USE_POSIX and self._name:
        from .resource_tracker import unregister
        _posixshmem.shm_unlink(self._name)
        unregister(self._name, "shared_memory")

和 (2) Windows 似乎在创建/使用它的进程停止后释放共享内存(请参阅评论 此处 和此处).这可能是 Python 没有明确处理这个的原因.

and (2) Windows seems to free up shared memory once the processeses that created/used it have stopped (see the comments here and here). This may be the cause for Python not handling this explicitly.

作为回应，我通过重复保存和重用相同的共享内存块而不取消链接来构建一个丑陋的解决方法.显然，这不是一个令人满意的解决方案，尤其是当所需内存块的大小动态变化时.

In response I have built an ugly workaround via saving and reusing the same shared memory block repeatedly without ever unlinking it. Obviously, this is not a satisfactory solution, especially if the sizes of the needed memory blocks change dynamically.

有没有办法手动释放 Windows 上的共享内存?

推荐答案

这是 multiprocessing 模块中的一个错误，报告为 问题 40882.有一个修复它的开放拉取请求，PR 20684，但显然它很慢合并.

This is a bug in the multiprocessing module, which was reported as Issue 40882. There is an open pull request that fixes it, PR 20684, though apparently it’s been slow to merge.

错误如下:在SharedMemory.__init__中，我们有调用 MapViewOfFile API 没有相应的 UnmapViewOfFile，并且 mmap 对象也不拥有它的所有权(它自己再次映射块).

The bug is as follows: in SharedMemory.__init__, we have an invocation of the MapViewOfFile API without a corresponding UnmapViewOfFile, and the mmap object does not take ownership of it either (it maps the block again on its own).

同时，您可以对 shared_memory 模块进行猴子修补，以便在构建 mmap 后添加缺失的 UnmapViewOfFile 调用.您可能不得不依赖 ctypes，因为 _winapi 模块不导出 UnmapViewOfFile，尽管导出了 MapViewOfFile (！).像这样的东西(未经测试):

In the meantime, you can monkey-patch the shared_memory module so that the missing UnmapViewOfFile call is added after mmap is constructed. You will probably have to rely on ctypes, as the _winapi module does not export UnmapViewOfFile, despite exporting MapViewOfFile (!). Something like this (not tested):

import ctypes, ctypes.wintypes
import multiprocessing, multiprocessing.shared_memory

UnmapViewOfFile = ctypes.windll.kernel32.UnmapViewOfFile
UnmapViewOfFile.argtypes = (ctypes.wintypes.LPCVOID,)
UnmapViewOfFile.restype = ctypes.wintypes.BOOL

def _SharedMemory_init(self, name=None, create=False, size=0):
    ... # copy from SharedMemory.__init__ in the original module
                try:
                    p_buf = _winapi.MapViewOfFile(
                        h_map,
                        _winapi.FILE_MAP_READ,
                        0,
                        0,
                        0
                    )
                finally:
                    _winapi.CloseHandle(h_map)
                try:
                    size = _winapi.VirtualQuerySize(p_buf)
                    self._mmap = mmap.mmap(-1, size, tagname=name)
                finally:
                    UnmapViewOfFile(p_buf)
    ... # copy from SharedMemory.__init__ in the original module

multiprocessing.shared_memory.SharedMemory.__init__ = _SharedMemory_init

将上述代码放入一个模块中，并记住在使用 multiprocessing 模块中的任何内容之前加载它.或者，您可以直接编辑 multiprocessing 模块目录中的 shared_memory.py 文件以包含所需的 UnmapViewOfFile 调用.这不是最干净的解决方案，但无论如何它都是暂时的(著名的遗言)；长期的解决方案是在上游修复这个问题(显然正在进行中).

Put the above code into a module and remember to load it before using anything from the multiprocessing module. Alternatively, you can directly edit the shared_memory.py file in the multiprocessing module's directory to contain the required UnmapViewOfFile call. This is not the cleanest solution, but it’s meant to be temporary anyway (famous last words); the long-term solution is to have this fixed upstream (as is apparently underway).

这篇关于“unlink()"在 Windows 上的 Python 共享内存中不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！