如何实现内存密集的python脚本进行测试 [英] How to implement a memory intensive python script for test

问题描述

我已经将cgroups规则应用于特定用户，并且我想测试从上述用户运行的程序的内存是否受到了预期的限制.我尝试使用以下脚本:

I've applied a cgroups rule to a specific user, and I'd like to test whether memory of the programs running from the above user has been limited as expected. I tried with the following script:

import string
import random

if __name__ == '__main__':
    d = {}
    i = 0;
    for i in range(0, 100000000):
        val = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(200)) # generate ramdom string of size 200
        d[i] = val
        if i % 10000 == 0:
            print i

当我通过ps命令监视该过程时，结果发现％MEM增加到4.8，并且在两个cgroups服务均打开和关闭时从未更改:

When I monitored the process via ps command, it turned out to be that the %MEM is increased to 4.8 and never changed when both cgroups service is on and off:

$ ps aux | grep mem_intensive.py
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
jason    11531 88.5  4.8 3312972 3191236 pts/0 R+   22:16   0:07 python mem_intensive.py

在这种情况下，总内存为62GB，因此其中的4.8％约为3GB.我将此限制设置为4GB，而该用户未运行任何其他进程.

In this scenario, total memory is 62GB, thus 4.8% of it about 3GB. I set the limit to be 4GB without any other processes running on this user.

那么谁能给我一些关于这个有问题的python脚本的想法?预先感谢.

So could anyone give me some idea about this problematic python script? Thanks in advance.

推荐答案

我已经对您的脚本进行了一些播放，尽管运行缓慢，它仍保持增长.瓶颈正在使用random.choice.如果您想快速填充内存，则产生随机性会对您不利.因此，仅使用固定字符串确实会很快耗尽内存.如果使用以下内容，则想查看其增长情况时，可能会在print之后扔一个time.sleep():

I've played a bit with your script, and it keeps growing, albeit slowly. The bottleneck is using random.choice. If you want to fill memory fast, generating randomness works against you. So just using fixed strings does exhaust the memory rather quickly. If using the following, while wanting to watch how it grows you'd probably throw a time.sleep() after your print:

if __name__ == '__main__':
    d = {}
    i = 0;
    for i in range(0, 100000000):
        d[i] = 'A'*1024
        if i % 10000 == 0:
            print(i)

更快地填充内存:

单线:

['A'*1024 for _ in xrange(0, 1024*1024*1024)]

这篇关于如何实现内存密集的python脚本进行测试的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！