在现有数组周围创建共享内存(python) [英] create shared memory around existing array (python)

问题描述

我在任何地方看到 python 的共享内存实现(例如在 multiprocessing 中)，创建共享内存总是分配新的内存.有没有办法创建共享内存对象并让它引用现有内存?目的是预先初始化数据值，或者更确切地说，如果我们已经有一个数组，则避免必须复制到新的共享内存中.根据我的经验，分配大型共享数组比将值复制到其中要快得多.

Everywhere I see shared memory implementations for python (e.g. in multiprocessing), creating shared memory always allocates new memory. Is there a way to create a shared memory object and have it refer to existing memory? The purpose would be to pre-initialize the data values, or rather, to avoid having to copy into the new shared memory if we already have, say, an array in hand. In my experience, allocating a large shared array is much faster than copying values into it.

推荐答案

简短的回答是否定的.

我是 Python 扩展的作者 posix_ipc1 和 sysv_ipc2.就像标准库中 Python 的 multiprocessing 模块一样，我的模块只是操作系统提供的工具的包装器，所以你真正需要知道的是操作系统在分配共享内存时允许什么.对于 SysV IPC 和 POSIX IPC，这有点不同，但在这种情况下，差异并不重要.(我认为多处理在可能的情况下使用 POSIX IPC.)

I'm the author of the Python extensions posix_ipc1 and sysv_ipc2. Like Python's multiprocessing module from the standard library, my modules are just wrappers around facilities provided by the operating system, so what you really need to know is what the OS allows when allocating shared memory. That differs a little for SysV IPC and POSIX IPC, but in this context the difference isn't really important. (I think multiprocessing uses POSIX IPC where possible.)

对于 SysV IPC，分配共享内存的操作系统级调用是 shmget().您可以在该调用的手册页上看到它不接受指向现有内存的指针；它总是为你分配新的内存.同一个调用 (shm_open()) 的 POSIX IPC 版本也是如此.POSIX IPC 很有趣，因为它实现共享内存看起来像一个内存映射文件，所以它的行为与 SysV IPC 有点不同.

For SysV IPC, the OS-level call to allocate shared memory is shmget(). You can see on that call's man page that it doesn't accept a pointer to existing memory; it always allocates new memory for you. Ditto for the POSIX IPC version of the same call (shm_open()). POSIX IPC is interesting because it implements shared memory to look like a memory mapped file, so it behaves a bit differently from SysV IPC.

无论是从 Python 还是 C 调用，都无法要求操作系统将现有的一块私有内存转换为共享内存.

Regardless, whether one is calling from Python or C, there's no option to ask the operating system to turn an existing piece of private memory into shared memory.

如果你仔细想想，你就会明白为什么.假设您可以将指向私有内存块的指针传递给 shmget() 或 shm_open().现在操作系统被困在将内存保持在原地的工作上，直到所有共享进程都用它完成.如果它在你的堆栈中间怎么办?突然间，您的堆栈中的这一大块无法分配，因为其他进程正在使用它.这也意味着当您的进程终止时，操作系统无法释放其所有内存，因为其中一些内存正在被其他进程使用.

If you think about it, you'll see why. Suppose you could pass a pointer to a chunk of private memory to shmget() or shm_open(). Now the operating system is stuck with the job of keeping that memory where it is until all sharing processes are done with it. What if it's in the middle of your stack? Suddenly this big chunk of your stack can't be allocated because other processes are using it. It also means that when your process dies, the OS can't release all its memory because some of it is now being used by other processes.

简而言之，没有提供您从 Python 要求的内容，因为底层操作系统调用不允许它，而底层操作系统调用不允许它(可能)，因为这对于操作系统.

In short, what you're asking for from Python isn't offered because the underlying OS calls don't allow it, and the underlying OS calls don't allow it (probably) because it would be really messy for the OS.

这篇关于在现有数组周围创建共享内存(python)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！