多处理内存使用情况和扭曲事件 [英] multiprocessing memory usage and twisted/gevents

问题描述

所以我的...错误...应用程序执行以下操作:

so my... err... app does the following:

• 在队列中听工作"
• 每台服务器(约3台服务器)产生大约100个工作线程，每个服务器都在队列中侦听
• 每个工作人员基本上都会做一些网络工作(ssh，snmp等)(I/O密集型)，然后搅动输出(非常cpu密集型)

我在multiprocessing下都能正常工作，效果很好.但是:每个工作人员正在使用比我想要的更多的内存(大约30MB RES，根据顶部450MB VIRT).所以我有两个问题:

i have it all working under multiprocessing and it works great. however: each worker is using way more memory than i would like (about 30MB RES, 450MB VIRT according to top). so i have two questions:

• 对我而言，确定开销如此之高的最佳方式是什么?我猜COW不能很好地工作...在进行多处理之前，我可以使用哪些模块来获取所有主线程内存的快照，以便我可以尝试减少初始占用空间?

• what is the best way for me to determine why the overhead is so high? i'm guessing COW isn't working too well... what modules could i use to get a snapshot of all of the main thread's memory prior to multiprocessing so i can try to reduce the initial footprint?

鉴于我的大多数进程都是cpu绑定的，将我的代码移植到gevent/twisted是否有好处?我想利用每台服务器的双十六进制核心.

given that most of my processes are cpu bound, would there be a benefit to port my code over to gevent/twisted? i would like to make use of the dual hex-cores of each server.

谢谢！

推荐答案

关于Pycon的精彩演讲解释了 Python中的内存使用情况.花了一个半小时的时间.

There was a great talk on Pycon which explains the subject of memory usage in python. It definitely a half an hour well spent.

最重要的是，要真正知道使用了多少内存，您不应该查看顶部输出，而是检查在运行100个工作线程之前和之后有多少可用内存.

The bottom line is that to really know how much memory is used you should not be looking at top output, but check how much memory you have free before and after running your 100 workers.

这篇关于多处理内存使用情况和扭曲事件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！