"apply_async"使“共享队列错误"静音. [英] `apply_async` silences "shared queue errors"
问题描述
请考虑以下示例:
from multiprocessing import Queue, Pool
def work(*args):
print('work')
return 0
if __name__ == '__main__':
queue = Queue()
pool = Pool(1)
result = pool.apply_async(work, args=(queue,))
print(result.get())
这将引发以下RuntimeError:
Traceback (most recent call last):
File "/tmp/test.py", line 11, in <module>
print(result.get())
[...]
RuntimeError: Queue objects should only be shared between processes through inheritance
但是有趣的是，仅当我尝试get结果时才引发异常，而不是在发生共享"时引发异常.在我实际上<共享>队列(并且从未执行过work！)的同时，注释相应的行将使错误静音.
所以这是我的问题:为什么仅在请求结果时才引发此异常，即使调用了apply_async方法时也不会引发这种异常，即使由于目标work函数从不被调用吗?
看起来异常发生在不同的进程中，并且仅当以请求结果的形式执行进程间通信时，该异常才可用于主进程.然后，我想知道为什么为什么 在分派到另一个进程之前不执行这样的检查.
(如果我在work和主要通信过程中都使用了队列，那么(无声地)会引入死锁.)
Python版本是3.5.2.
我已阅读以下问题:
- 在Python中的进程之间共享许多队列
- 如何将队列引用传递给pool.map_async()管理的功能?
- 在多个进程之间共享结果队列
- Python多处理:RuntimeError:队列对象仅应通过继承在进程之间共享"
- Python在进程之间共享锁
此行为是由multiprocessing.Pool的设计引起的.
在内部，当您调用apply_async时，您将作业放在Pool调用队列中，然后取回AsyncResult对象，该对象使您可以使用get检索计算结果.
然后，另一个线程负责使您的工作腌制.在该线程中，发生了RuntimeError，但是您已经从对async_apply的调用中返回了.因此，它会将AsyncResult中的结果设置为例外，当您调用get时会引发该结果.
尝试使用 concurrent.futures 具有明确的未来对象，而IMO是一种更好的处理故障的设计，可以让您查询未来对象是否发生故障而无需调用get函数.
Consider the following example:
from multiprocessing import Queue, Pool
def work(*args):
print('work')
return 0
if __name__ == '__main__':
queue = Queue()
pool = Pool(1)
result = pool.apply_async(work, args=(queue,))
print(result.get())
This raises the following RuntimeError:
Traceback (most recent call last):
File "/tmp/test.py", line 11, in <module>
print(result.get())
[...]
RuntimeError: Queue objects should only be shared between processes through inheritance
But interestingly the exception is only raised when I try to get the result, not when the "sharing" happens. Commenting the corresponding line silences the error while I actually did share the queue (and work is never executed!).
So here goes my question: Why is this exception only raised when the result is requested, and not when the apply_async method is invoked even though the error seems to be recognized because the target work function is never called?
It looks like the exception occurs in a different process and can only be made available to the main process when inter-process communication is performed in form of requesting the result. Then, however, I'd like to know why such checks are not performed before dispatching to the other process.
(If I used the queue in both work and the main process for communication then this would (silently) introduce a deadlock.)
Python version is 3.5.2.
I have read the following questions:
- Sharing many queues among processes in Python
- How do you pass a Queue reference to a function managed by pool.map_async()?
- Sharing a result queue among several processes
- Python multiprocessing: RuntimeError: "Queue objects should only be shared between processes through inheritance"
- Python sharing a lock between processes
This behavior results from the design of the multiprocessing.Pool.
Internally, when you call apply_async, you put your job in the Pool call queue and then get back a AsyncResult object, which allow you to retrieve your computation result using get.
Another thread is then in charge of pickling your work. In this thread, the RuntimeError happens but you already returned from the call to async_apply. Thus, it sets the results in AsyncResult as the exception, which is raised when you call get.
This behavior using some kind of future results is better understood when you try using concurrent.futures, which have explicit future objects and, IMO, a better design to handle failures, has you can query the future object for failure without calling the get function.
这篇关于"apply_async"使“共享队列错误"静音.的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
