在子进程已经启动之后授予对共享内存的访问权限 [英] Giving access to shared memory after child processes have already started
问题描述
如果仅在生成子进程后才可用，则如何授予子进程访问共享内存中的数据的权限(使用
How do I give child processes access to data in shared memory if the data is only available after the child processes have been spawned (using multiprocessing.Process)?
我知道 multiprocessing.sharedctypes.RawArray ，但我不知道如何让子进程访问在进程启动后创建的RawArray.
I am aware of multiprocessing.sharedctypes.RawArray, but I can't figure out how to give my child processes access to a RawArray that is created after the processes have already started.
数据是由父进程生成的，并且数据量事先未知.
The data is generated by the parent process, and the amount of data is not known in advance.
如果不是 GIL ，我将使用线程代替来完成此任务稍微简单一点.不能选择使用非CPython实现.
If not for the GIL I'd be using threading instead which will make this task a little simpler. Using a non-CPython implementation is not an option.
Looking under the hood of muliprocessing.sharedctypes, it looks like shared ctype objects are allocated using mmaped memory.
因此，这个问题实际上可以归结为:如果在子进程产生后由父级调用mmap()，子进程可以访问匿名映射的内存吗?
So this question really boils down to: Can a child process access an anonymously mapped memory if mmap() was called by the parent after the child process was spawned?
这有点符合这方面的要求问题，只是在我的情况下，mmap()的调用者是父进程，而不是子进程.
That's somewhat in the vein of what's being asked in this question, except that in my case the caller of mmap() is the parent process and not the child process.
我创建了自己的RawArray版本，该版本在引擎盖下使用shm_open().只要标识符(tag)匹配，结果共享的ctypes数组就可以与任何进程共享.
I created my own version of RawArray that uses shm_open() under the hood. The resulting shared ctypes array can be shared with any process as long as the identifier (tag) matches.
请参见此答案详细信息和示例.
See this answer for details and an example.
推荐答案
您的问题听起来很像 posix_ipc或sysv_ipc模块，它们公开了POSIX或SysV API，用于共享内存，信号量和消息队列.那里的功能矩阵包括从他提供的模块中挑选的极好的建议.
Your problem sounds like a perfect fit for the posix_ipc or sysv_ipc modules, which expose either the POSIX or SysV APIs for shared memory, semaphores, and message queues. The feature matrix there includes excellent advice for picking amongst the modules he provides.
匿名mmap(2)区域的问题在于，您无法轻松地与其他进程共享它们-如果它们是文件支持的，这很容易，但是如果您实际上不需要该文件，那么，感觉很傻.如果在C语言中，您可以在clone(2)系统调用中使用CLONE_VM标志，但是我不想在可能对内存安全性做出假设的语言解释器中尝试使用它. (即使在C语言中也有些危险，因为五年后的维护程序员可能 也会被CLONE_VM行为震惊.)
The problem with anonymous mmap(2) areas is that you cannot easily share them with other processes -- if they were file-backed, it'd be easy, but if you don't actually need the file for anything else, it feels silly. You could use the CLONE_VM flag to the clone(2) system call if this were in C, but I wouldn't want to try using it with a language interpreter that probably makes assumptions about memory safety. (It'd be a little dangerous even in C, as maintenance programmers five years from now might also be shocked by the CLONE_VM behavior.)
但是SysV和较新的POSIX共享内存映射甚至允许不相关的进程按标识符从共享内存中进行连接和分离，因此您要做的就是与创建映射的进程共享使用标识符的进程中的标识符，并与使用该映射的进程共享该标识符. ，然后在映射中操作数据时，它们可同时供所有进程使用，而无需任何额外的解析开销. shm_open(3)函数返回一个int，在以后对ftruncate(2)和mmap(2)的调用中用作文件描述符，因此其他进程可以使用共享内存段，而无需在文件系统中创建文件.即使使用该内存的所有进程都已退出，该内存也将保留. (也许对Unix有点奇怪，但它很灵活.)
But the SysV and newer POSIX shared memory mappings allow even unrelated processes to attach and detach from shared memory by identifier, so all you need to do is share the identifier from the processes that create the mappings with the processes that consume the mappings, and then when you manipulate data within the mappings, they are available to all processes simultaneously without any additional parsing overhead. The shm_open(3) function returns an int that is used as a file descriptor in later calls to ftruncate(2) and then mmap(2), so other processes can use the shared memory segment without a file being created in the filesystem -- and this memory will persist even if all processes using it have exited. (A little strange for Unix, perhaps, but it is flexible.)
这篇关于在子进程已经启动之后授予对共享内存的访问权限的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
