Python多重处理永远不会加入 [英] Python multiprocessing never joins

问题描述

我正在使用multiprocessing，特别是使用Pool来剥离几个线程"来完成我所进行的一系列缓慢的工作.但是，由于某些原因，即使所有孩子似乎都已死亡，我也无法重新加入主线程.

I'm using multiprocessing, and specifically a Pool to spin off a couple of 'threads' to do a bunch of slow jobs that I have. However, for some reason, I can't get the main thread to rejoin, even though all of the children appear to have died.

已解决:该问题的答案似乎是仅启动多个Process对象，而不是使用Pool.原因尚不十分清楚，但我怀疑剩余的过程是池的管理者，并且在过程完成时不会死.如果还有其他人有这个问题，那就是答案.

Resolved: It appears the answer to this question is to just launch multiple Process objects, rather than using a Pool. It's not abundantly clear why, but I suspect the remaining process is a manager for the pool and it's not dying when the processes finish. If anyone else has this problem, this is the answer.

主线程

pool = Pool(processes=12,initializer=thread_init)
for x in xrange(0,13):
    pool.apply_async(thread_dowork)
pool.close()
sys.stderr.write("Waiting for jobs to terminate\n")
pool.join()

xrange(0,13)比进程数多一个，因为我认为自己的工作量减少了一个，而且一个进程没有找到工作，所以没有死，所以我想强迫它执行找工作我也尝试了12种.

The xrange(0,13) is one more than the number of processes because I thought I had an off by one, and one process wasn't getting a job, so wasn't dying and I wanted to force it to take a job. I have tried it with 12 as well.

多处理功能

def thread_init():
    global log_out
    log_out = open('pool_%s.log'%os.getpid(),'w')
    sys.stderr = log_out
    sys.stdout = log_out
    log_out.write("Spawned")
    log_out.flush()
    log_out.write(" Complete\n")
    log_out.flush()


def thread_dowork():
    log_out.write("Entered function\n")
    log_out.flush()
    #Do Work
    log_out.write("Exiting ")
    log_out.flush()
    log_out.close()
    sys.exit(0)

所有12个子级的日志文件的输出为:

The output of the logfiles for all 12 children is:

Spawned
Complete
Entered function
Exiting

主线程打印正在等待作业终止"，然后坐在那里.

The main thread prints 'Waiting for jobs to terminate', and then just sits there.

top仅显示该脚本的一个副本(我相信主要的副本). htop显示两个副本，其中一个是自上而下的副本，另一个是其他副本.根据其PID，它也不是所有子级.

top shows only one copy of the script (the main one I believe). htop shows two copies, one of which is the one from top, and the other one of which is something else. Based on its PID, it's none of the children either.

有人知道我不知道的东西吗?

Does anyone know something I don't?

推荐答案

我确实没有答案，但是我阅读了Apply_async的文档，似乎与您提出的问题背道而驰...

I don't really have an answer but I read the docs for Apply_async and it seems counter to your stated problem...

回调应立即完成，因为否则该线程将 处理结果将被阻止.

Callbacks should complete immediately since otherwise the thread which handles the results will get blocked.

我不熟悉Pool，但在我看来，您可以通过

I'm not familiar with the Pool but it seems to me that your use-case could easily be handled by this recipe on Python Module of the Week

这篇关于Python多重处理永远不会加入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！