使用subprocess.Popen的Python内存分配错误 [英] Python memory allocation error using subprocess.Popen

问题描述

我正在做一些生物信息学工作.我有一个python脚本，它在某一时刻调用一个程序来执行一个昂贵的过程(序列对齐..使用大量的计算能力和内存).我用subprocess.Popen来称呼它.当我在一个测试用例上运行它时，它会完成并完成得很好.但是，当我在完整文件上运行它时(对于不同的输入集，它必须多次执行此操作)，它死了.子进程抛出:

I am doing some bioinformatics work. I have a python script that at one point calls a program to do an expensive process (sequence alignment..uses a lot of computational power and memory). I call it using subprocess.Popen. When I run it on a testcase, it completes and finishes fine. However, when I run it on the full file, where it would have to do this multiple times for different sets of inputs, it dies. Subprocess throws:

OSError: [Errno 12] Cannot allocate memory

我在此处和此处存在类似问题，但我不确定它们是否适用于我的情况.

I found a few links here and here and here to similar problems, but I'm not sure that they apply in my case.

默认情况下，序列比对器将尝试请求51000M的内存.它并不总是使用那么多，但是可能会用到.加载并处理了完整的输入后，那么多的内容将不可用.但是，限制它请求或尝试以较低的数量(在运行时可用)使用的数量仍然会给我同样的错误.我也尝试过使用shell = True和同样的东西运行.

By default, the sequence aligner will try to request 51000M of memory. It doesn't always use that much, but it might. With the full input loaded and processed, that much is not available. However, capping the amount it requests or will attempt to use at a lower amount that might be available when running still gives me the same error. I've also tried running with shell=True and same thing.

这已经困扰了我几天了.感谢您的帮助.

This has been bugging me for a few days now. Thanks for any help.

扩展回溯:

File "..../python2.6/subprocess.py", line 1037, in _execute_child
    self.pid=os.fork()
OSError: [Errno 12] Cannot allocate memory

引发错误.

Edit2:在64位ubuntu 10.4上的python 2.6.4中运行

Running in python 2.6.4 on 64 bit ubuntu 10.4

推荐答案

我为OP感到非常抱歉. 6年后，没有人提到这是Unix中一个非常普遍的问题，实际上与python或生物信息学无关.调用os.fork()会暂时使父进程的内存加倍(父进程的内存必须对子进程可用)，然后再将其全部扔掉以执行exec().虽然并不总是实际复制此内存，但是系统必须具有足够的内存才能复制它，因此，如果您的父进程正在使用系统内存的一半以上，那么您甚至可以对"wc -l"进行子处理，您将遇到内存错误.

I feel really sorry for the OP. 6 years later and no one mentioned that this is a very common problem in Unix, and actually has nothing to do with python or bioinformatics. A call to os.fork() temporarily doubles the memory of the parent process (the memory of the parent process must be available to the child process), before throwing it all away to do an exec(). While this memory isn't always actually copied, the system must have enough memory to allow for it to be copied, and thus if you're parent process is using more than half of the system memory and you subprocess out even "wc -l", you're going to run into a memory error.

解决方案是使用posix_spawn或在脚本开头创建所有子进程，而内存消耗较低，然后在父进程完成占用大量内存的事情之后再使用它们.

The solution is to use posix_spawn, or create all your subprocesses at the beginning of the script, while memory consumption is low, then use them later on after the parent process has done it's memory-intensive thing.

使用keyworks"os.fork"和"memory"进行的google搜索将显示该主题上的一些Stack Overflow帖子，这些帖子可以进一步解释发生了什么事情:)

A google search using the keyworks "os.fork" and "memory" will show several Stack Overflow posts on the topic that can further explain what's going on :)

这篇关于使用subprocess.Popen的Python内存分配错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！