无法在Python 3.5中子类化多处理队列 [英] Cannot subclass multiprocessing Queue in Python 3.5

问题描述

我最终的目标是将stdout从几个子流程重定向到一些队列，并将它们打印到某个地方(也许在一个小的GUI中).

My eventual goal is to redirect the stdout from several subprocesses to some queues, and print those out somewhere (maybe in a little GUI).

第一步是将Queue子类化为行为类似于stdout的对象.但这就是我被困住的地方.在Python v3.5中，将多处理Queue子类化似乎是不可能的.

The first step is to subclass Queue into an object that behaves much like the stdout. But that is where I got stuck. Subclassing the multiprocessing Queue seems impossible in Python v3.5.

# This is a Queue that behaves like stdout
# Unfortunately, doesn't work in Python 3.5   :-(
class StdoutQueue(Queue):
    def __init__(self,*args,**kwargs):
        Queue.__init__(self,*args,**kwargs, ctx='')

    def write(self,msg):
        self.put(msg)

    def flush(self):
        sys.__stdout__.flush()

我在以下帖子中找到了此代码段(当时可能还不存在Python 3.5):

I found this snippet in the following post (probably Python 3.5 did not yet exist at that moment): Python multiprocessing redirect stdout of a child process to a Tkinter Text

在Python v3.5中，当将多处理Queue类子类化时，会偶然发现奇怪的错误消息.我发现了两个描述该问题的错误报告:

In Python v3.5 you stumble on strange error messages when subclassing the multiprocessing Queue class. I found two bug reports describing the issue:

https://bugs.python.org/issue21367

https://bugs.python.org/issue19895

我有2个问题:

1. 假设我要坚持使用Python v3.5-切换到以前的版本并不是一种选择.我可以使用哪种解决方法以某种方式对多处理队列进行子类化?
2. 如果我升级到Python v3.6，该错误是否仍然存在?

---

当您尝试对此处找到的Queue类进行子类化时，存在一个已知问题:

There is a known issue when you try to subclass the Queue class found in here:

from multiprocessing import Queue    # <- known issue: you cannot subclass
                                     #    this Queue class, because it is
                                     #    not a genuine python class.

但是以下方法应该起作用:

But the following should work:

from multiprocessing.queues import Queue   # <- from this Queue class, you
                                           #    should be able to make a
                                           #    subclass. But Python 3.5
                                           #    refuses :-(

可悲的是，即使在Python v3.5中也不起作用.您收到以下错误:

Sadly, even that doesn't work in Python v3.5. You get the following error:

    C:\Users\..\myFolder > python myTest.py

        Traceback (most recent call last):
            File "myTest.py", line 49, in <module>
              q = StdoutQueue()
            File "myTest.py", line 22, in __init__
              super(StdoutQueue,self).__init__(*args,**kwargs)
        TypeError: __init__() missing 1 required keyword-only argument: 'ctx'

---

谢谢达斯·科蒂克(Darth Kotik)解决了这个问题！这是完整的代码，并使用他的解决方案进行了更新.现在可以了.

Thank you Darth Kotik for solving the problem! Here is the complete code, updated with his solution. Now it works.

import sys
import time
import multiprocessing as mp
import multiprocessing.queues as mpq
from threading import Thread
from tkinter import *

'''-------------------------------------------------------------------'''
'''                SUBCLASSING THE MULTIPROCESSING QUEUE              '''
'''                                                                   '''
'''         ..and make it behave as a general stdout io               '''
'''-------------------------------------------------------------------'''
# The StdoutQueue is a Queue that behaves like stdout.
# We will subclass the Queue class from the multiprocessing package
# and give it the typical stdout functions.
#
# (1) First issue
# Subclassing multiprocessing.Queue or multiprocessing.SimpleQueue
# will not work, because these classes are not genuine
# python classes.
# Therefore, you need to subclass multiprocessing.queues.Queue or
# multiprocessing.queues.SimpleQueue . This issue is known, and is not
# the reason for asking this question. But I mention it here, for
# completeness.
#
# (2) Second issue
# There is another problem that arises only in Python V5 (and beyond).
# When subclassing multiprocessing.queues.Queue, you have to provide
# a 'multiprocessing context'. Not doing that, leads to an obscure error
# message, which is in fact the main topic of this question. Darth Kotik
# solved it.
# His solution is visible in this code:
class StdoutQueue(mpq.Queue):

    def __init__(self,*args,**kwargs):
        ctx = mp.get_context()
        super(StdoutQueue, self).__init__(*args, **kwargs, ctx=ctx)

    def write(self,msg):
        self.put(msg)

    def flush(self):
        sys.__stdout__.flush()


'''-------------------------------------------------------------------'''
'''                           TEST SETUP                              '''
'''-------------------------------------------------------------------'''

# This function takes the text widget and a queue as inputs.
# It functions by waiting on new data entering the queue, when it
# finds new data it will insert it into the text widget.
def text_catcher(text_widget,queue):
    while True:
        text_widget.insert(END, queue.get())

def test_child(q):
    # This line only redirects stdout inside the current process
    sys.stdout = q
    # or sys.stdout = sys.__stdout__ if you want to print the child to the terminal
    print('child running')

def test_parent(q):
    # Again this only redirects inside the current (main) process
    # commenting this like out will cause only the child to write to the widget
    sys.stdout = q
    print('parent running')
    time.sleep(0.5)
    mp.Process(target=test_child,args=(q,)).start()

if __name__ == '__main__':
    gui_root = Tk()
    gui_txt = Text(gui_root)
    gui_txt.pack()
    q = StdoutQueue()
    gui_btn = Button(gui_root, text='Test', command=lambda:test_parent(q),)
    gui_btn.pack()

    # Instantiate and start the text monitor
    monitor = Thread(target=text_catcher,args=(gui_txt,q))
    monitor.daemon = True
    monitor.start()

    gui_root.mainloop()

推荐答案

>>> import multiprocessing
>>> type(multiprocessing.Queue)
<class 'method'>
AttributeError: module 'multiprocessing' has no attribute 'queues'
>>> import multiprocessing.queues
>>> type(multiprocessing.queues.Queue)
<class 'type'>

因此您可以看到multiprocessing.Queue只是multiprocessing.queues.Queue类的构造方法.如果要创建子类，只需class MyQueue(multiprocessing.queues.Queue)

So as you can see multiprocessing.Queue is just constructor method for multiprocessing.queues.Queue class. If you want to make a child class just do class MyQueue(multiprocessing.queues.Queue)

您可以在此处查看此方法的来源

编辑: 好的.我现在知道了你的问题.如您在上面的链接中所见，multiprocessing.Queue将ctx参数传递给Queue.所以我设法自己通过__init__方法来使其工作.我不完全理解BaseContext对象应该在哪里获取_name属性，因此我手动传递了它.

EDIT: Okay. I got your problem now. As you can see on a link above, multiprocessing.Queue passes ctx argument to Queue. So I managed to get it working by doing it myself in __init__ method. I don't completely inderstand where BaseContext object supposed to get _name attribute, so I passed it manually.

def __init__(self,*args,**kwargs):
    from multiprocessing.context import BaseContext
    ctx = BaseContext()
    ctx._name = "Name"
    super(StdoutQueue,self).__init__(*args,**kwargs, ctx=ctx)

EDIT2 :事实证明文档具有有关上下文此处.因此，不用像我一样手动创建它，您就可以

EDIT2: Turned out docs have some information about context here. So instead of manually creating it like I did you can do

import multiprocessing
ctx = multiprocessing.get_context()

它将在设置了_name的情况下创建适当的上下文(在特定情况下为"fork")，您可以将其传递到队列中.

It will create proper context with _name set (to 'fork' in your particular case) and you can pass it to your queue.

这篇关于无法在Python 3.5中子类化多处理队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！