在Python中生成不确定的守护进程 [英] Indefinite daemonized process spawning in Python

问题描述

我正在尝试构建一个启动其他完全独立进程的Python守护程序。

I'm trying to build a Python daemon that launches other fully independent processes.

通常的想法是针对给定的shell命令，每隔几秒钟轮询一次并确保该命令的 k 个实例正在运行。我们保存一个pidfiles目录，当我们轮询时，将删除其pids不再运行的pidfiles，并启动（并为其创建pidfiles），但是我们需要许多进程来获取它们 k 。

The general idea is for a given shell command, poll every few seconds and ensure that exactly k instances of the command are running. We keep a directory of pidfiles, and when we poll we remove pidfiles whose pids are no longer running and start up (and make pidfiles for) however many processes we need to get to k of them.

子进程也需要完全独立，这样，如果父进程死了，子进程就不会被杀死。从我阅读的内容来看，似乎没有办法通过子流程模块来完成此操作。为此，我使用了此处提到的代码段：

The child processes also need to be fully independent, so that if the parent process dies the children won't be killed. From what I've read, it seems there is no way to do this with the subprocess module. To this end, I used the snippet mentioned here:

http://code.activestate.com/recipes/66012-fork-a-daemon-process-on-unix/

我做了几处必要的修改（您会在所附的代码段中看到注释行）：

I made a couple necessary modifications (you'll see the lines commented out in the attached snippet):

1. 原始父级进程无法退出，因为我们需要启动器守护程序无限期地持久。


2. 子进程需要以与父进程相同的cwd开始。

这是我的衍生工具fn和一个测试：

Here's my spawn fn and a test:

import os
import sys
import subprocess
import time

def spawn(cmd, child_cwd):
    """
    do the UNIX double-fork magic, see Stevens' "Advanced 
    Programming in the UNIX Environment" for details (ISBN 0201563177)
    http://www.erlenstar.demon.co.uk/unix/faq_2.html#SEC16
    """
    try: 
        pid = os.fork() 
        if pid > 0:
            # exit first parent
            #sys.exit(0) # parent daemon needs to stay alive to launch more in the future
            return
    except OSError, e: 
        sys.stderr.write("fork #1 failed: %d (%s)\n" % (e.errno, e.strerror))
        sys.exit(1)

    # decouple from parent environment
    #os.chdir("/") # we want the children processes to 
    os.setsid() 
    os.umask(0) 

    # do second fork
    try: 
        pid = os.fork() 
        if pid > 0:
            # exit from second parent
            sys.exit(0) 
    except OSError, e: 
        sys.stderr.write("fork #2 failed: %d (%s)\n" % (e.errno, e.strerror))
        sys.exit(1) 

    # redirect standard file descriptors
    sys.stdout.flush()
    sys.stderr.flush()
    si = file('/dev/null', 'r')
    so = file('/dev/null', 'a+')
    se = file('/dev/null', 'a+', 0)
    os.dup2(si.fileno(), sys.stdin.fileno())
    os.dup2(so.fileno(), sys.stdout.fileno())
    os.dup2(se.fileno(), sys.stderr.fileno())

    pid = subprocess.Popen(cmd, cwd=child_cwd, shell=True).pid

    # write pidfile       
    with open('pids/%s.pid' % pid, 'w') as f: f.write(str(pid))
    sys.exit(1)

def mkdir_if_none(path):
    if not os.access(path, os.R_OK):
        os.mkdir(path)

if __name__ == '__main__':
    try:
        cmd = sys.argv[1]
        num = int(sys.argv[2])
    except:
        print 'Usage: %s <cmd> <num procs>' % __file__
        sys.exit(1)
    mkdir_if_none('pids')
    mkdir_if_none('test_cwd')

    for i in xrange(num):
        print 'spawning %d...'%i
        spawn(cmd, 'test_cwd')
        time.sleep(0.01) # give the system some breathing room

在这种情况下，事情似乎运行良好，即使父进程被杀死，子进程也会继续存在。但是，我仍然遇到原始父级的生成限制。在大约650次生成（不是同时完成，即子代完成）之后，父进程因错误而窒息：

In this situation, things seem to work fine, and the child processes persist even when the parent is killed. However, I'm still running into a spawn limit on the original parent. After ~650 spawns (not concurrently, the children have finished) the parent process chokes with the error:

spawning 650...
fork #2 failed: 35 (Resource temporarily unavailable)

有没有办法重写我的生成函数，以便可以无限期生成这些独立的子进程？谢谢！

Is there any way to rewrite my spawn function so that I can spawn these independent child processes indefinitely? Thanks!

推荐答案

感谢您的进程列表我愿意说这是因为您遇到了许多基本限制之一：

Thanks to your list of processes I'm willing to say that this is because you have hit one of a number of fundamental limitations:

• rlimit nproc 允许给定用户执行的最大进程数-请参阅 setrlimit（2）， bash（1）内置的 ulimit 和 /etc/security/limits.conf 有关每个用户进程限制的详细信息。


• rlimit nofile 允许给定进程打开的文件描述符的最大数量一旦。 （每个新进程可能在 parent 中为孩子的 stdin ， stdout 和 stderr 描述符。）


• 系统范围内的最大进程数；参见 / proc / sys / kernel / pid_max 。


• 系统范围内打开文件的最大数量；请参阅 / proc / sys / fs / file-max 。

• rlimit nproc maximum number of processes a given user is allowed to execute -- see setrlimit(2), the bash(1) ulimit built-in, and /etc/security/limits.conf for details on per-user process limits.
• rlimit nofile maximum number of file descriptors a given process is allowed to have open at once. (Each new process probably creates three new pipes in the parent, for the child's stdin, stdout, and stderr descriptors.)
• System-wide maximum number of processes; see /proc/sys/kernel/pid_max.
• System-wide maximum number of open files; see /proc/sys/fs/file-max.

因为您没有收获死去的孩子，这些资源中的许多资源开放的时间比他们应有的长。您的 second 子代已由 init（8）正确处理-他们的父母已去世，因此将其重新父母为 init（8）和 init（8）将在它们之后清除（ wait（2）

Because you're not reaping your dead children, many of these resources are held open longer than they should. Your second children are being properly handled by init(8) -- their parent is dead, so they are re-parented to init(8), and init(8) will clean up after them (wait(2)) when they die.

但是，您的程序有责任在第一批孩子出生后进行清理。 C程序通常为 SIGCHLD 的 signal（7）处理程序安装一个调用 wait（2）的处理程序。 或 waitpid（2）来获得孩子的退出状态，从而从内核内存中删除其条目。

However, your program is responsible for cleaning up after the first set of children. C programs typically install a signal(7) handler for SIGCHLD that calls wait(2) or waitpid(2) to reap the children's exit status and thus remove its entries from the kernel's memory.

但是脚本中的信号处理有点烦人。如果您可以将 SIGCHLD 信号处置显式设置为 SIG_IGN ，则内核将知道您对退出不感兴趣

But signal handling in a script is a bit annoying. If you can set the SIGCHLD signal disposition to SIG_IGN explicitly, the kernel will know that you are not interested in the exit status and will reap the children for you_.

尝试添加：

import signal
signal.signal(signal.SIGCHLD, signal.SIG_IGN)

请注意，我不知道这对 Subprocess 的作用。可能不满意。如果是这种情况，那么您需要安装信号处理程序进行调用 wait（2）为您。

Note that I don't know what this does for Subprocess. It might not be pleased. If that is the case, then you'll need to install a signal handler to call wait(2) for you.

这篇关于在Python中生成不确定的守护进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！