了解Python分叉和内存分配错误 [英] Understanding Python fork and memory allocation errors

问题描述

我有一个占用大量内存的Python应用程序(介于数百MB到几GB之间).
我有几个非常小的Linux可执行文件，例如主要应用程序需要运行.

I have a memory intensive Python application (between hundreds of MB to several GB).
I have a couple of VERY SMALL Linux executables the main application needs to run, e.g.

child = Popen("make html", cwd = r'../../docs', stdout = PIPE, shell = True)
child.wait()

当我使用subprocess.Popen运行这些外部实用程序(一次，在长时间的主进程运行结束时)时，有时会得到OSError: [Errno 12] Cannot allocate memory.
我不明白为什么...所请求的过程很小！
系统具有足够的内存以容纳更多外壳.

When I run these external utilities (once, at the end of the long main process run) using subprocess.Popen I sometimes get OSError: [Errno 12] Cannot allocate memory.
I don't understand why... The requested process is tiny!
The system has enough memory for many more shells.

我正在使用Linux(Ubuntu 12.10，64位)，所以我猜子进程会调用Fork.
并且Fork分叉了我现有的进程，因此使内存消耗翻了一番，并且失败了?
写时复制"发生了什么?

I'm using Linux (Ubuntu 12.10, 64 bits), so I guess subprocess calls Fork.
And Fork forks my existing process, thus doubling the amount of memory consumed, and fails??
What happened to "copy on write"?

我可以在没有fork的情况下(或至少在没有复制内存的情况下-重新开始)生成新进程吗?

Can I spawn a new process without fork (or at least without copying memory - starting fresh)?

相关:

fork()，vfork()，exec( )和clone()

fork()&内存分配行为

Python子进程.Popen出现OSError错误:[Errno 12]在一段时间后无法分配内存

使用subprocess.Popen的Python内存分配错误

推荐答案

似乎不会出现真正的解决方案(即使用vfork的子流程的替代实现).那么可爱的骇客如何呢?在您的过程开始时，产生一个随身携带的，占用很小内存的从属服务器，准备产生您的子进程，并在整个主进程的生命周期中保持与之的开放通信.

It doesn't appear that a real solution will be forthcoming (i.e. an alternate implementation of subprocess that uses vfork). So how about a cute hack? At the beginning of your process, spawn a slave that hangs around with a small memory footprint, ready to spawn your subprocesses, and keep open communication to it throughout the life of the main process.

以下是使用rfoo的示例( http://code.google.com/p/rfoo/)和一个名为rfoosocket的名为unix的套接字(您显然可以使用rfoo支持的其他连接类型，或另一个RPC库):

Here's an example using rfoo (http://code.google.com/p/rfoo/) with a named unix socket called rfoosocket (you could obviously use other connection types rfoo supports, or another RPC library):

服务器:

import rfoo
import subprocess

class MyHandler(rfoo.BaseHandler):
    def RPopen(self, cmd):
        c = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
        c.wait()
        return c.stdout.read()

rfoo.UnixServer(MyHandler).start('rfoosocket')

客户:

import rfoo

# Waste a bunch of memory before spawning the child. Swap out the RPC below
# for a straight popen to show it otherwise fails. Tweak to suit your
# available system memory.
mem = [x for x in range(100000000)]

c = rfoo.UnixConnection().connect('rfoosocket')

print rfoo.Proxy(c).RPopen('ls -l')

如果您需要与生成的子流程进行实时来回协同处理，则此模型可能无法正常工作，但您可能可以破解它.您可能希望清理可用的args.根据您的特定需求传递给Popen，但这应该相对简单.

If you need real-time back and forth coprocess interaction with your spawned subprocesses this model probably won't work, but you might be able to hack it in. You'll presumably want to clean up the available args that can be passed to Popen based on your specific needs, but that should all be relatively straightforward.

您还应该直接在客户端启动时启动服务器，并管理要在退出时清理的套接字文件(或端口).

You should also find it straightforward to launch the server at the start of the client, and to manage the socket file (or port) to be cleaned up on exit.

这篇关于了解Python分叉和内存分配错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！