Python进程池非守护进程? [英] Python Process Pool non-daemonic?
问题描述
是否可以创建非守护进程的python池?我希望一个池能够调用其中包含另一个池的函数.
我想要这个,因为守护进程无法创建进程.具体来说,它将导致错误:
AssertionError: daemonic processes are not allowed to have children
例如,请考虑以下情况:function_a
具有运行function_b
的池,而该池具有运行function_c
的池.该功能链将失败,因为function_b
正在守护进程中运行,并且守护进程无法创建进程.
multiprocessing.pool.Pool
类在其__init__
方法中创建辅助进程,使其成为守护进程并启动它们,并且无法重新设置它们在False
之前将daemon
属性赋给False
(此后将不再允许).但是您可以创建自己的multiprocesing.pool.Pool
子类(multiprocessing.Pool
只是一个包装函数),并替换您自己的multiprocessing.Process
子类,该子类始终是非守护进程的,以用于工作进程. /p>
这是如何执行此操作的完整示例.重要的部分是顶部的两个类NoDaemonProcess
和MyPool
,最后在您的MyPool
实例上调用pool.close()
和pool.join()
.
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
import multiprocessing
# We must import this explicitly, it is not imported by the top-level
# multiprocessing module.
import multiprocessing.pool
import time
from random import randint
class NoDaemonProcess(multiprocessing.Process):
# make 'daemon' attribute always return False
def _get_daemon(self):
return False
def _set_daemon(self, value):
pass
daemon = property(_get_daemon, _set_daemon)
# We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool
# because the latter is only a wrapper function, not a proper class.
class MyPool(multiprocessing.pool.Pool):
Process = NoDaemonProcess
def sleepawhile(t):
print("Sleeping %i seconds..." % t)
time.sleep(t)
return t
def work(num_procs):
print("Creating %i (daemon) workers and jobs in child." % num_procs)
pool = multiprocessing.Pool(num_procs)
result = pool.map(sleepawhile,
[randint(1, 5) for x in range(num_procs)])
# The following is not really needed, since the (daemon) workers of the
# child's pool are killed when the child is terminated, but it's good
# practice to cleanup after ourselves anyway.
pool.close()
pool.join()
return result
def test():
print("Creating 5 (non-daemon) workers and jobs in main process.")
pool = MyPool(5)
result = pool.map(work, [randint(1, 5) for x in range(5)])
pool.close()
pool.join()
print(result)
if __name__ == '__main__':
test()
Would it be possible to create a python Pool that is non-daemonic? I want a pool to be able to call a function that has another pool inside.
I want this because deamon processes cannot create process. Specifically, it will cause the error:
AssertionError: daemonic processes are not allowed to have children
For example, consider the scenario where function_a
has a pool which runs function_b
which has a pool which runs function_c
. This function chain will fail, because function_b
is being run in a daemon process, and daemon processes cannot create processes.
The multiprocessing.pool.Pool
class creates the worker processes in its __init__
method, makes them daemonic and starts them, and it is not possible to re-set their daemon
attribute to False
before they are started (and afterwards it's not allowed anymore). But you can create your own sub-class of multiprocesing.pool.Pool
(multiprocessing.Pool
is just a wrapper function) and substitute your own multiprocessing.Process
sub-class, which is always non-daemonic, to be used for the worker processes.
Here's a full example of how to do this. The important parts are the two classes NoDaemonProcess
and MyPool
at the top and to call pool.close()
and pool.join()
on your MyPool
instance at the end.
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
import multiprocessing
# We must import this explicitly, it is not imported by the top-level
# multiprocessing module.
import multiprocessing.pool
import time
from random import randint
class NoDaemonProcess(multiprocessing.Process):
# make 'daemon' attribute always return False
def _get_daemon(self):
return False
def _set_daemon(self, value):
pass
daemon = property(_get_daemon, _set_daemon)
# We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool
# because the latter is only a wrapper function, not a proper class.
class MyPool(multiprocessing.pool.Pool):
Process = NoDaemonProcess
def sleepawhile(t):
print("Sleeping %i seconds..." % t)
time.sleep(t)
return t
def work(num_procs):
print("Creating %i (daemon) workers and jobs in child." % num_procs)
pool = multiprocessing.Pool(num_procs)
result = pool.map(sleepawhile,
[randint(1, 5) for x in range(num_procs)])
# The following is not really needed, since the (daemon) workers of the
# child's pool are killed when the child is terminated, but it's good
# practice to cleanup after ourselves anyway.
pool.close()
pool.join()
return result
def test():
print("Creating 5 (non-daemon) workers and jobs in main process.")
pool = MyPool(5)
result = pool.map(work, [randint(1, 5) for x in range(5)])
pool.close()
pool.join()
print(result)
if __name__ == '__main__':
test()
这篇关于Python进程池非守护进程?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!