如何在python进程中限制内存使用 [英] How to limit memory usage within a python process
问题描述
我在具有16GB Ram和64位OS的Linux机器上运行Python 2.7.我编写的python脚本可能会将过多的数据加载到内存中,这使计算机的运行速度降低到我什至无法杀死进程的地步.
I run Python 2.7 on a Linux machine with 16GB Ram and 64 bit OS. A python script I wrote can load too much data into memory, which slows the machine down to the point where I cannot even kill the process any more.
虽然我可以通过以下方式限制内存:
While I can limit memory by calling:
ulimit -v 12000000
在运行脚本之前在我的shell中,我想在脚本本身中包含一个限制选项.我到处看的地方,resource
模块被引用为具有与ulimit
相同的功能.但是打电话:
in my shell before running the script, I'd like to include a limiting option in the script itself. Everywhere I looked, the resource
module is cited as having the same power as ulimit
. But calling:
import resource
_, hard = resource.getrlimit(resource.RLIMIT_DATA)
resource.setrlimit(resource.RLIMIT_DATA, (12000, hard))
在我的脚本开始时完全不执行任何操作.即使将值设置为低至12000,也不会使该过程崩溃.我尝试使用RLIMIT_STACK
进行同样的操作,并且结果相同.奇怪的是,打电话:
at the beginning of my script does absolutely nothing. Even setting the value as low as 12000 never crashed the process. I tried the same with RLIMIT_STACK
, as well with the same result. Curiously, calling:
import subprocess
subprocess.call('ulimit -v 12000', shell=True)
什么也没做.
我做错了什么?我在网上找不到任何实际用法示例.
What am I doing wrong? I couldn't find any actual usage examples online.
对于任何好奇的人,使用subprocess.call
不起作用,因为它创建了一个(令人惊讶的惊喜!)新进程,该进程与当前python程序运行所在的进程无关.
edit: For anyone who is curious, using subprocess.call
doesn't work because it creates a (surprise, surprise!) new process, which is independent of the one the current python program runs in.
推荐答案
resource.RLIMIT_VMEM
是与ulimit -v
对应的资源
resource.RLIMIT_VMEM
is the resource corresponding to ulimit -v
.
RLIMIT_DATA
仅影响brk/sbrk
系统调用,而
RLIMIT_DATA
only affects brk/sbrk
system calls while newer memory managers tend to use mmap
instead.
要注意的第二件事是 ulimit
/
The second thing to note is that ulimit
/setrlimit
only affects the current process and its future children.
关于AttributeError: 'module' object has no attribute 'RLIMIT_VMEM'
消息: resource
模块文档提到了这种可能性:>
Regarding the AttributeError: 'module' object has no attribute 'RLIMIT_VMEM'
message: the resource
module docs mention this possibility:
此模块不会尝试掩盖平台差异-符号 未针对平台定义的版本将无法通过以下模块使用 该平台.
This module does not attempt to mask platform differences — symbols not defined for a platform will not be available from this module on that platform.
根据链接的 bash
ulimit
源链接以上,如果未定义RLIMIT_VMEM
,则使用RLIMIT_AS
.
According to the bash
ulimit
source linked to above, it uses RLIMIT_AS
if RLIMIT_VMEM
is not defined.
这篇关于如何在python进程中限制内存使用的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!