使用Python共享内存的分段错误 [英] Segmentation Fault using Python Shared Memory

问题描述

函数 store_in_shm 将numpy数组写入共享内存，而第二个函数 read_from_shm 使用相同共享内存空间中的数据创建numpy数组并返回numpy数组

The function store_in_shm writes a numpy array to the shared memory while the second function read_from_shm creates a numpy array using data in the same shared memory space and returns the numpy array.

但是，在Python 3.8中运行代码会产生以下分段错误:

However, running the code in Python 3.8 gives the following segmentation error:

zsh:分段错误python foo.py

zsh: segmentation fault python foo.py

为什么从函数 read_from_shm 内部访问numpy数组没有问题，但是在函数外部再次访问numpy数组时出现分段错误?

Why is there no problem accessing the numpy array from inside the function read_from_shm, but a segmentation error appears when accessing the numpy array again outside of the function?

输出:

From read_from_shm(): [0 1 2 3 4 5 6 7 8 9]
zsh: segmentation fault  python foo.py
% /Users/athena/opt/anaconda3/envs/test/lib/python3.8/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

foo.py

import numpy as np
from multiprocessing import shared_memory

def store_in_shm(data):
    shm = shared_memory.SharedMemory(name='foo', create=True, size=data.nbytes)
    shmData = np.ndarray(data.shape, dtype=data.dtype, buffer=shm.buf)
    shmData[:] = data[:]
    shm.close()
    return shm

def read_from_shm(shape, dtype):
    shm = shared_memory.SharedMemory(name='foo', create=False)
    shmData = np.ndarray(shape, dtype, buffer=shm.buf)
    print('From read_from_shm():', shmData)
    return shmData

if __name__ == '__main__':
    data = np.arange(10)
    shm = store_in_shm(data)
    shmData = read_from_shm(data.shape, data.dtype)
    print('From __main__:', shmData)    # no seg fault if we comment this line
    shm.unlink()

推荐答案

基本上，问题似乎在于底层的mmap文件(由 read_from_shm 中的 shm 拥有)函数返回时，当 shm 被垃圾回收时，)被关闭.然后 shmData 指向它，这就是您得到段错误(用于表示封闭的mmap)的地方.这似乎是已知错误，但可以通过保留对 shm 的引用来解决.

Basically the problem seems to be that the underlying mmap'ed file (owned by shm within read_from_shm) is being closed when shm is garbage collected when the function returns. Then shmData refers back to it, which is where you get the segfault (for referring to a closed mmap) This seems to be a known bug, but it can be solved by keeping a reference to shm.

此外，所有 SharedMemory 实例都希望被 close()'实例化，而其中一个实例恰好在被 unlink()'ed实例化时不再需要.如果您自己不调用 shm.close()，则会如前所述在GC处调用它，而在Windows上，如果它是当前唯一打开"的，则会在Windows上调用它.共享内存文件将被删除.当您在 store_in_shm 内调用 shm.close()时，您会引入OS依赖项，因为在Windows上，数据将被删除，而MacOS和Linux将一直保留到 unlink 被调用.

Additionally all SharedMemory instances want to be close()'d with exactly one of them being unlink()'ed when it is no longer necessary. If you don't call shm.close() yourself, it will be called at GC as mentioned, and on Windows if it is the only one currently "open" the shared memory file will be deleted. When you call shm.close() inside store_in_shm, you introduce an OS dependency as on windows the data will be deleted, and MacOS and Linux, it will retain until unlink is called.

最后，尽管这没有出现在您的代码中，但在从中访问数据的地方，目前仍然存在另一个问题独立的进程(而不是子进程)可能会过早地删除基础mmap. SharedMemory 是一个非常新的库，希望所有的缺点都能很快解决.

Finally though this doesn't appear in your code, another problem currently exists where accessing data from independent processes (rather than child processes) can similarly delete the underlying mmap too soon. SharedMemory is a very new library, and hopefully all the kinks will work out soon.

您可以重写给定的示例，以保留对"second"的引用. shm ，只需使用其中一个即可取消链接:

You can re-write the given example to retain a reference to the "second" shm and just use either one to unlink:

import numpy as np
from multiprocessing import shared_memory

def store_in_shm(data):
    shm = shared_memory.SharedMemory(name='foo', create=True, size=data.nbytes)
    shmData = np.ndarray(data.shape, dtype=data.dtype, buffer=shm.buf)
    shmData[:] = data[:]
    #there must always be at least one `SharedMemory` object open for it to not
    #  be destroyed on Windows, so we won't `shm.close()` inside the function,
    #  but rather after we're done with everything.
    return shm

def read_from_shm(shape, dtype):
    shm = shared_memory.SharedMemory(name='foo', create=False)
    shmData = np.ndarray(shape, dtype, buffer=shm.buf)
    print('From read_from_shm():', shmData)
    return shm, shmData #we need to keep a reference of shm both so we don't
                        #  segfault on shmData and so we can `close()` it.

if __name__ == '__main__':
    data = np.arange(10)
    shm1 = store_in_shm(data)
    #This is where the *Windows* previously reclaimed the memory resulting in a 
    #  FileNotFoundError because the tempory mmap'ed file had been released.
    shm2, shmData = read_from_shm(data.shape, data.dtype)
    print('From __main__:', shmData)
    shm1.close() 
    shm2.close()
    #on windows "unlink" happens automatically here whether you call `unlink()` or not.
    shm2.unlink() #either shm1 or shm2

这篇关于使用Python共享内存的分段错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！