Python中混合处理和线程处理的状态 [英] Status of mixing multiprocessing and threading in Python
问题描述
就问题6721而言，在Linux的同一python应用程序中同时使用多处理线程和用户线程的最佳实践或变通方法是什么，应该在fork上清除python标准库中的锁?
What are best practices or work-arounds for using both multiprocessing and user threads in the same python application in Linux with respect to Issue 6721, Locks in python standard library should be sanitized on fork?
为什么我两个都需要?我使用子进程进行繁重的计算，从而产生的数据结构结果太大而无法通过队列返回-而是必须将它们立即存储到磁盘.由一个单独的线程监视这些子进程中的每个进程似乎很有效，因此完成后，线程可以处理将大(例如，多GB)数据读回进程的IO，在该进程中需要结果进行进一步的计算.与其他子进程的结果结合. 子进程会间歇性地挂起，我只是(在经过很多头部撞击之后)发现是通过使用日志记录模块引起"的.其他人在这里记录了该问题:
Why do I need both? I use child processes to do heavy computation that produce data structure results that are much too large to return through a queue -- rather they must be immediately stored to disk. It seemed efficient to have each of these child processes monitored by a separate thread, so that when finished, the thread could handle the IO of reading the large (eg multi GB) data back into the process where the result was needed for further computation in combination with the results of other child processes. The children processes would intermittently hang, which I just (after much head pounding) found was 'caused' by using the logging module. Others have documented the problem here:
https://twiki.cern.ch/twiki/bin/view/Main/PythonLoggingThreadingMultiprocessingIntermixedStudy
指向这个显然未解决的python问题:python标准库中的锁应在fork上进行清理； http://bugs.python.org/issue6721
which points to this apparently unsolved python issue: Locks in python standard library should be sanitized on fork; http://bugs.python.org/issue6721
对我一直难以找到的问题感到震惊，我回答:
Alarmed at the difficulty I had tracking this down, I answered:
带有小心"的相当无益的建议，并指向上面的链接.
with the rather unhelpful suggestion to 'Be careful' and links to the above.
但是冗长的讨论是:问题6721建议在同一应用程序中同时使用多处理(或os.fork)和用户线程是一个错误".由于对问题的了解有限，我在讨论中发现太多分歧，无法得出在同一应用程序中同时使用多处理和线程处理的解决方法或策略.我的紧迫问题是通过禁用日志记录来解决的，但是我在父进程和子进程中都创建了少量的其他(显式)锁，并怀疑我正在为进一步的间歇性死锁做好准备.
But the lengthy discussion re: Issue 6721 suggests that it is a 'bug' to use both multiprocessing (or os.fork) and user threads in the same application. With my limited understanding of the problem, I find too much disagreement in the discussion to conclude what are the work-arounds or strategies for using both multiprocessing and threading in the same application. My immediate problem was solved by disabling logging, but I create a small handful of other (explicit) locks in both parent and child processes, and suspect I am setting myself up for further intermittent deadlocks.
您能否提出实用的建议，以避免在python(2.7,3.2,3.3)应用程序中使用锁和/或日志记录模块时使用线程和多处理时出现死锁?
Can you give practical recommendations to avoid deadlocks while using locks and/or the logging module while using threading and multiprocessing in a python (2.7,3.2,3.3) application?
推荐答案
如果在程序中仍然只有一个线程的情况下分叉其他进程(即在产生工作线程之前从主线程分叉)，将是安全的. ).
You will be safe if you fork off additional processes while you still have only one thread in your program (that is, fork from main thread, before spawning worker threads).
您的用例看起来像您甚至不需要多处理模块；您可以使用子流程(甚至更简单的类似于os.system的调用).
Your use case looks like you don't even need multiprocessing module; you can use subprocess (or even simpler os.system-like calls).
另请参见从线程内分叉是否安全? ?
这篇关于Python中混合处理和线程处理的状态的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
