多处理模块显示与主进程相同的每个子进程的内存. [英] Multiprocessing module showing memory for each child process same as Main process.
问题描述
我正在使用python的多处理模块，对此有一些困惑.
I am using multiprocessing module of python and have some confusions regarding the same.
基本上，我最初在Main进程中存储一些数据，如top命令所示，大约为16GB(主内存大小).我已经将这些数据存储为全局变量.
Basically, I store some data initially in the Main process, and that is around 16GB (main memory size) as shown in the top command. I have stored these data as global Variables.
然后对这些数据进行多重处理，并分别进行相应的处理和不同的处理.
Then multiprocessing is done on this data, and processed accordingly and differently accordingly.
现在我看到正在发生多处理，即所有进程都有自己的CPU利用率，但是所有进程的内存各为16 GB.为什么?是不是应该使用我通过引用全局变量而通过传递的相同内存..请注意一些想法.
Now I see that multiprocessing is happening i.e. all processes has its own CPU utilization, but the memory of all the processes in 16 GB each.. why so.?? Isn't it should use same memory that I send through pass by reference of global variables.. Please some thoughts.
top命令的输出如下:-
The output of top command is as follows.:-
PID用户PR NI VIRT RES SHR S％CPU％MEM TIME +命令 13908管理20 0 16.7g 16g 848 R 100.0 17.3 0:32.92 python
13429管理20 0 16.7g 16g 3336 S 0.0 17.3 15:06.97 python
13910管理20 0 16.7g 16g 848 R 100.3 17.3 0:32.94 python
13911管理20 0 16.7g 16g 840 R 100.0 17.3 0:33.02 python
13912管理20 0 16.7g 16g 836 R 99.6 17.3 0:33.00 python
13907管理20 0 16.7g 16g 796 R 100.0 17.3 0:33.06 python
13909管理20 0 16.7g 16g 796 R 99.6 17.3 0:32.93 python
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13908 admin 20 0 16.7g 16g 848 R 100.0 17.3 0:32.92 python
13429 admin 20 0 16.7g 16g 3336 S 0.0 17.3 15:06.97 python
13910 admin 20 0 16.7g 16g 848 R 100.3 17.3 0:32.94 python
13911 admin 20 0 16.7g 16g 840 R 100.0 17.3 0:33.02 python
13912 admin 20 0 16.7g 16g 836 R 99.6 17.3 0:33.00 python
13907 admin 20 0 16.7g 16g 796 R 100.0 17.3 0:33.06 python
13909 admin 20 0 16.7g 16g 796 R 99.6 17.3 0:32.93 python
推荐答案
multiprocessing模块产生的每个进程都在单独的地址空间中.创建新进程后，原始进程拥有的所有物理和虚拟内存至少在逻辑上独立于新进程，但最初每个新进程都是该进程的精确副本(很好，请参见脚注).老的.因此，每个虚拟机都将具有与原始虚拟机相同的虚拟大小(16.7 GB).
Each process spawned by the multiprocessing module is in a separate address space. All physical and virtual memory that the original process had is at least logically independent of the new ones after the new ones are created, but initially each new process is an exact duplicate (well, see footnote) of the old. Thus, each will have the same virtual size (16.7 GB) as the original.
使用写时复制"来尽可能共享实际的基础物理页面.当各种副本运行并对其虚拟内存进行更改时，内核将根据需要复制基础物理页面.从未写入的内存可以在所有副本之间共享.因此，即使每个进程似乎占用大量RAM，但实际上并非如此.不过，如果您要写入其中的大部分数据(即，如果每个单独的进程更改了16 GB的大部分数据)，那么它们都将具有单独的副本，并使用更多的物理RAM.
Actual underlying physical pages are shared as much as possible, using "copy-on-write". As the various copies run and make changes to their virtual memory, the kernel will copy the underlying physical page as needed. Memory that is never written-to can be shared between all the copies. So even though each process appears to be chewing up a lot of RAM, they aren't, really. If you write to most of it, though—i.e., if each separate process changes most of the 16 GB of data—then they will all have separate copies, and use much more physical RAM.
multiprocessing模块确实提供了一些共享数据的方法(请参见 http://docs.python.org/library/multiprocessing.html )，如果您希望他们共享修改(但请考虑锁定的工作方式；请参阅文档).
脚注:在进行fork或clone系统调用之后，原始副本和克隆副本之间有一个微小的区别:原始副本获取克隆副本的ID，而克隆副本获取数字零.
The multiprocessing module does offer some methods of sharing data (see the "shared memory" section in http://docs.python.org/library/multiprocessing.html) if you want them to share modifications (but then think about how the locking works; see the documentation).
footnote: There's one tiny difference between the original and the clone, after a fork or clone system call: the original gets back the ID of the clone, and the clone gets back the number zero.
这篇关于多处理模块显示与主进程相同的每个子进程的内存.的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
