Python多重处理导致许多僵尸进程 [英] Python Multiprocessing leading to many zombie processes

问题描述

我一直在使用工人池来实现python的多处理库.我实现了以下代码

I have been implementing python's multiprocessing library using a pool of workers. I implemented the following code

import main1
t1 = time.time()
p = Pool(cores) 
result = p.map(main1, client_list[client])
if result == []:
    return []
p.close()
p.join()
print "Time taken in performing request:: ", time.time()-t1
return shorted(result)

但是，在运行了一段时间后，我的应用程序得到了很多正在运行的后台进程.这是为我的应用程序执行ps aux后的快照

However, after running the process for a while, I get lot of running background processes of my app. Here is a snapshot after doing ps aux for my app

现在，我已经阅读了很多关于stackoverflow的类似问题，例如 Python多处理终止进程.但是我想知道我的代码可能出什么毛病. 我无法在main1函数中共享我的所有代码，但是我将整个代码块放在try catch块中，以避免主代码中的错误可能导致僵尸进程的情况.

Now, I have read a lot of similar questions on stackoverflow like how to kill zombie processes created by multiprocessing module? which calls for using .join() which I have already implemented and I learned how to kill all these processes from here Python Multiprocessing Kill Processes. But I want to know what possibly could go wrong with my code. I won't able to share all of my code in the main1 function but I have put the entire code block in try catch block to avoid cases where an error in the main code could lead to zombie processes.

def main1((param1, param2, param3)):
    try:
       resout.append(some_data) //resout in case of no error
    except:
        print traceback.format_exc()
        resout = []  //sending empty resout in case of error
    return resout

我对并行编程和调试问题的概念还很陌生，事实证明它很棘手.任何帮助将不胜感激.

I'm still very new to the concept of parallel programming and debugging issues with it is turning out to be tricky.Any help will be greatly appreciated.

推荐答案

通常，最常见的问题是创建了池但未关闭该池.

Usually the most common problem is that the pool is created but it is not closed.

我知道保证池关闭的最好方法是使用try/finally子句:

The best way I know to guarantee that the pool is closed is to use a try/finally clause:

try:
    pool = Pool(ncores)
    pool.map(yourfunction, arguments)
finally:
    pool.close()
    pool.join()

如果您不想与multiprocessing纠缠，我写了一个名为parmap的简单程序包，该程序包包装了多处理程序，使我(甚至可能是您)的生活更加轻松.

If you don't want to struggle with multiprocessing, I wrote a simple package named parmap that wraps multiprocessing to make my life (and potentially yours) easier.

pip install parmap

import parmap
parmap.map(yourfunction, arguments)

在parmap使用情况部分中:

From the parmap usage section:

• 简单的并行示例:

• Simple parallel example:

import parmap
y1 = [myfunction(x, argument1, argument2) for x in mylist]
y2 = parmap.map(myfunction, mylist, argument1, argument2)
y1 == y2

遍历元组列表:

Iterating over a list of tuples:

# You want to do:
z = [myfunction(x, y, argument1, argument2) for (x,y) in mylist]
z = parmap.starmap(myfunction, mylist, argument1, argument2)


# You want to do:
listx = [1, 2, 3, 4, 5, 6]
listy = [2, 3, 4, 5, 6, 7]
param = 3.14
param2 = 42
listz = []
for (x, y) in zip(listx, listy):
    listz.append(myfunction(x, y, param1, param2))
# In parallel:
listz = parmap.starmap(myfunction, zip(listx, listy), param1, param2)

这篇关于Python多重处理导致许多僵尸进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！