父出口上的Python多处理队列 [英] Python Multiprocessing Queue on Parent Exit

问题描述

我要问的要点是，当父进程(在这种情况下为守护程序)被杀死时，多处理队列会发生什么情况.

The gist of my question is what happens to a multiprocessing queue when the parent (a daemon in this circumstance) is killed.

我有一个后台运行的守护进程，它将子进程的作业排队:

I have a daemon that runs in the background which queues up jobs for child processes:

class manager(Daemon):
    def run(self):
        someQueue = MP.Queue()

        someChild = MP.Process(target=someCode, args=(someArgs))
        someChild.start()
        ...

如果管理器被杀死(假设它不是试图使用someQueue并因此如文档中所述损坏了它)，是否有办法恢复队列中的数据?

If the manager is killed (assuming it wasn't trying to use someQueue and therefore corrupted it as mentioned in the documentation), is there anyway to recover the data in the Queue?

我看到的两个理论解决方案是在退出此子进程之前清理someChild中的someQueue.另外转储队列，以便我可以在经理退出时恢复队列的状态，这也可以解决我的问题.但是，在实施这两种方法之前，最好将其推向正确的方向.

Two theoretical solutions I see are cleaning up the someQueue in someChild before exiting this child process. Also dumping the queues so that I could restore the state of the queues when the manager exited would also solve my problem. However, before implementing either it would be nice to get nudged in the right direction.

谢谢

推荐答案

听起来您想要持久/可靠的排队.我相信multiprocessing.Queue类是通过管道实现的(就像您通过popen()调用获得的那样)，因此数据是相对瞬态的，您可能必须做一些操作系统级的技巧才能获取内容.您可能会考虑编写自己的持久队列类，该类使用文件系统文件(假设操作系统和文件系统支持锁定)来存储队列内容.然后，您可以提供所需的所有分析工具来检查队列并恢复未处理的数据.

It sounds like you want persistent/reliable queuing. I believe the multiprocessing.Queue class is implemented with pipes (just like you would get with a popen() call), so the data is relatively transient and you'd probably have to do some OS-level trickery to grab the contents. You might look into writing your own persistent queue class that uses a filesystem file (assuming your OS and filesystem support locking) to store the queue contents. Then you can provide all of the analysis tools you desire to inspect the queue and recover unprocessed data.

这篇关于父出口上的Python多处理队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！