守护进程内部的Python多处理池 [英] Python multiprocessing pool inside daemon process

问题描述

我对此问题提出了一个问题，但没有得到足够彻底的答案来解决该问题(很可能是由于在解释我的问题时缺乏严谨性，这是我要纠正的问题):python多处理守护程序中的僵尸进程

I opened up a question for this problem and did not get a thorough enough answer to solve the issue (most likely due to a lack of rigor in explaining my issues which is what I am attempting to correct): Zombie process in python multiprocessing daemon

我正在尝试实现一个python守护程序，该守护程序使用工作池来使用Popen执行命令.我从 http://www.jejik.com/articles/借来了基本守护程序2007/02/a_simple_unix_linux_daemon_in_python/

I am trying to implement a python daemon that uses a pool of workers to executes commands using Popen. I have borrowed the basic daemon from http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/

我只更改了init，daemonize(或同等的start)和stop方法.这是对init方法的更改:

I have only changed the init, daemonize (or equally the start) and stop methods. Here are the changes to the init method:

def __init__(self, pidfile):
#, stdin='/dev/null', stdout='STDOUT', stderr='STDOUT'):
    #self.stdin = stdin
    #self.stdout = stdout
    #self.stderr = stderr
    self.pidfile = pidfile
    self.pool = Pool(processes=4)

我没有设置stdin，stdout和stderr，以便可以使用print语句调试代码.另外，我尝试将这个池移到几个地方，但这是唯一不产生异常的地方.

I am not setting stdin, stdout and stderr so that I can debug the code with print statements. Also, I have tried moving this pool around to a few places but this is the only place that does not produce exceptions.

这是对daemonize方法的更改:

def daemonize(self):
    ...

    # redirect standard file descriptors
    #sys.stdout.flush()
    #sys.stderr.flush()
    #si = open(self.stdin, 'r')
    #so = open(self.stdout, 'a+')
    #se = open(self.stderr, 'a+', 0)
    #os.dup2(si.fileno(), sys.stdin.fileno())
    #os.dup2(so.fileno(), sys.stdout.fileno())
    #os.dup2(se.fileno(), sys.stderr.fileno())

    print self.pool

    ...

同一件事，我没有重定向io，以便可以调试.使用这里的打印，以便我可以检查池的位置.

Same thing, I am not redirecting io so that I can debug. The print here is used so that I can check the pools location.

stop方法更改:

def stop(self):
    ...

    # Try killing the daemon process
    try:
        print self.pool
        print "closing pool"
        self.pool.close()
        print "joining pool"
        self.pool.join()
        print "set pool to None"
        self.pool = None
        while 1:
            print "kill process"
            os.kill(pid, SIGTERM)

    ...

这里的想法是我不仅需要终止进程，还需要清理池. self.pool = None只是为了解决无法解决的问题而进行的随机尝试.起初我以为这是僵尸儿童的问题，当我在os.kill(pid, SIGTERM)的while循环中放入self.pool.close()和self.pool.join()时，就出现了问题.这是在我决定开始通过print self.pool查看游泳池位置之前.完成此操作后，我相信守护程序启动和停止时的池都不相同.这是一些输出:

Here the idea is that I not only need to kill the process but also clean up the pool. The self.pool = None is just a random attempt to solve the issues which didn't work. At first I thought this was a problem with zombie children which was occurring when I had the self.pool.close() and self.pool.join() inside the while loop with the os.kill(pid, SIGTERM). This is before I decided to start looking at the pool location via the print self.pool. After doing this, I believe the pools are not the same when the daemon starts and when it stops. Here is some output:

me@pc:~/pyCode/jobQueue$ sudo ./jobQueue.py start
<multiprocessing.pool.Pool object at 0x1c543d0>
me@pc:~/pyCode/jobQueue$ sudo ./jobQueue.py stop
<multiprocessing.pool.Pool object at 0x1fb7450>
closing pool
joining pool
set pool to None
kill process
kill process
... [ stuck in infinite loop]

对象的不同位置向我暗示它们不是同一池，并且其中一个可能是僵尸?

The different locations of the objects suggest to me that they are not the same pool and that one of them is probably the zombie?

在CTRL+C之后，这是我从ps aux|grep jobQueue中获得的信息:

After CTRL+C, here is what I get from ps aux|grep jobQueue:

root     21161  0.0  0.0  50384  5220 ?        Ss   22:59   0:00 /usr/bin/python ./jobQueue.py start
root     21162  0.0  0.0      0     0 ?        Z    22:59   0:00 [jobQueue.py] <defunct>
me       21320  0.0  0.0   7624   940 pts/0    S+   23:00   0:00 grep --color=auto jobQueue

我尝试将self.pool = Pool(processes=4)移到许多不同的地方.如果将其移至start()' or daemonize()methods, print self.pool`，将抛出一个异常，指出它是NoneType.此外，该位置似乎可以改变将弹出的僵尸进程的数量.

I have tried moving the self.pool = Pool(processes=4) to a number of different places. If it is moved to the start()' ordaemonize()methods,print self.pool` will throw an exception saying that it is NoneType. In addition, the location seems to change the number of zombie process that will pop up.

当前，我还没有添加通过工作程序运行任何东西的功能.我的问题似乎与正确设置工作人员池完全相关.我将不胜感激任何导致解决此问题的信息，或提供有关创建守护程序服务的建议，该服务使用工作人员池使用Popen执行一系列命令.由于我还没走那么远，所以我不知道我面临的挑战是什么.我想我可能只需要编写自己的池，但是如果有一个不错的技巧可以使池在这里工作，那就太好了.

Currently, I have not added the functionality to run anything via the workers. My problem seems completely related to setting up the pool of workers correctly. I would appreciate any information that leads to solving this issue or advice about creating a daemon service that uses a pool of workers to execute a series of commands using Popen. Since I haven't gotten that far, I do not know what challenges I face ahead. I am thinking I might just need to write my own pool but if there is a nice trick to make the pool work here, it would be amazing.

推荐答案

解决方案是将self.pool = Pool(process=4)放在daemonize方法的最后一行.否则，池最终会迷失在某个地方(也许在fork中).然后可以在run方法内部访问该池，该方法由您要守护的应用程序重载.但是，无法在stop方法中访问该池，否则将导致NoneType异常.我相信有一个更优雅的解决方案，但是这可行，这就是我目前所拥有的.如果我想在池仍在运行时stop失败，我将不得不向run添加附加功能并添加某种形式的消息，但我目前对此并不关心.

The solution is to put the self.pool = Pool(process=4) as the last line of the daemonize method. Otherwise the pool ends up getting lost somewhere (perhaps in the forks). Then the pool can be access inside the run method which is overloaded by the application you wish to daemonize. However, the pool cannot be accessed in the stop method and to do so would lead to NoneType exceptions. I believe there is a more elegant solution but this works and it is all I have for now. If I want the stop to fail when the pool is still in action, I will have to add additional functionality to run and some form of message but I am not currently concerned with this.

这篇关于守护进程内部的Python多处理池的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！