Python中的子流程模块出现资源暂时不可用的错误 [英] Resource temporarily unavailable error with subprocess module in Python

问题描述

在Python中，我产生了一个gnuplot进程以从数据集中生成gif图像.

In Python, I spawn a gnuplot process to generate gif images from a data set.

from subprocess import Popen, PIPE
def gnuplotter(...)
    p = Popen([GNUPLOT], shell=False, stdin=PIPE, stdout=PIPE)
    p.stdin.write(r'set terminal gif;')
    ...
    p.stdin.write(contents)
    p.stdout.close()

当我一次使用gnuplotter()时它工作正常，但是当我多次启动该进程时，出现了Resource temporarily unavailable错误.

It works fine when I use gnuplotter() one time, but when I launch the process multiple times, I got Resource temporarily unavailable error.

for i in range(54):
    gnuplotter(i, ... 

  File "/Users/smcho/code/PycharmProjects/contextAggregator/aggregation_analyzer/aggregation_analyzer/gnuplotter.py", line 48, in gnuplotter
    p = Popen([GNUPLOT], shell=False, stdin=PIPE, stdout=PIPE)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1205, in _execute_child
    self.pid = os.fork()
OSError: [Errno 35] Resource temporarily unavailable

出什么问题了，如何在喷出另一个之前关闭gnuplot进程?

What's wrong, and how can I close gnuplot process before spewing another one?

推荐答案

pid号，打开的文件描述符，内存是有限的资源.

pid numbers, open file descriptors, memory are limited resources.

fork(2)手册说当errno.EAGAIN应该发生的时间:

[EAGAIN]  The system-imposed limit on the total number of processes under
          execution would be exceeded.  This limit is configuration-dependent.

[EAGAIN]  The system-imposed limit MAXUPRC () on the total number of processes
          under execution by a single user would be exceeded.

要更轻松地重现该错误，可以在程序的开头添加:

To reproduce the error more easily, you could add at the start of your program:

import resource

resource.setrlimit(resource.RLIMIT_NPROC, (20, 20))

问题可能是所有子进程都处于活动状态，因为您尚未调用p.stdin.close()，并且在重定向到管道时，gnuplot的stdin可能会被完全缓冲，即，gnuplot进程可能会被卡住以等待输入.和/或您的应用程序使用了太多的文件描述符(文件描述符在Python 2.7上默认由子进程继承)而没有释放它们.

The issue might be that all child processes are alive because you haven't called p.stdin.close() and gnuplot's stdin might be fully buffered when redirected to a pipe i.e., gnuplot processes might be stuck awaiting input. And/or your application uses too many file descriptors (file descriptors are inherited by child processes by default on Python 2.7) without releasing them.

如果输入不依赖于输出并且输入的大小受到限制，请使用.communicate():

If input doesn't depend on the output and the input is limited in size then use .communicate():

from subprocess import Popen, PIPE, STDOUT

p = Popen("gnuplot", stdin=PIPE, stdout=PIPE, stderr=PIPE,
          close_fds=True, # to avoid running out of file descriptors
          bufsize=-1, # fully buffered (use zero (default) if no p.communicate())
          universal_newlines=True) # translate newlines, encode/decode text
out, err = p.communicate("\n".join(['set terminal gif;', contents]))

.communicate()写入所有输入并读取所有输出(同时，因此没有死锁)，然后关闭p.stdin，p.stdout，p.stderr(即使输入很小并且gnuplot的一侧已完全缓冲； EOF刷新缓冲区)并等待该过程完成(没有僵尸).

.communicate() writes all input and reads all output (concurrently, so there is no deadlock) then closes p.stdin, p.stdout, p.stderr (even if input is small and gnuplot's side is fully buffered; EOF flushes the buffer) and waits for the process to finish (no zombies).

Popen在其轮询的构造函数中调用_cleanup()所有已知子进程的退出状态，即，即使您不调用p.wait()，也不应有太多僵尸进程(死态但处于未读状态).

Popen calls _cleanup() in its constructor that polls exit status of all known subprocesses i.e., even if you won't call p.wait() there shouldn't be many zombie processes (dead but with unread status).

这篇关于Python中的子流程模块出现资源暂时不可用的错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！