Matplotlib和多处理RuntimeError [英] Matplotlib and multiprocessing RuntimeError

问题描述

我正在尝试同时使用multiprocessing和matplotlib.

I'm trying to use multiprocessing and matplotlib together.

我正在创建标准的Pool，使用apply_async添加工作，并使用apply_async的回调函数更新GUI，该回调函数在Pool的父进程上运行(我已通过os.getpid()进行了验证).例子:

I'm creating a standard Pool, adding work with apply_async and updating the GUI with apply_async's callback function, which runs on the Pool's parent process (I verified this with os.getpid()). Example :

from pylab import *
from numpy import *
from numpy.random import random
from multiprocessing import Pool

# Output image
global out_all
out_all = zeros((256, 256))

# Only does something to in_image, doesn't access anything else
def do_work(in_image):
    for x in xrange(100000):
        out_image = in_image[::-1, ::-1]
    return out_image

# Update the output image and display if needed
def do_update(out_image):
    global out_all
    print ("Updating")
    out_all += out_image
    clf()
    imshow(out_all)
    show()

# Input images (close enough to what I do as well)
work = [random((256, 256)) for f in range(20)]

# Don't block when showing something
ion()

# Do the work
print "Starting pool"
pool = Pool()
for o in work:
    pool.apply_async(do_work, [o], callback=do_update).get()
pool.close()
pool.join()
print "Stopping pool"

# Block
ioff()
show()
print "Done"

处理本身可以正常工作，进程在pool.join()上确实被破坏了，但是Matplotlib(我猜是TK，我猜想)在我后来尝试做某事时抱怨，甚至只是退出程序:

The processing itself works fine, the processes are really destroyed on pool.join(), but Matplotlib (and TK, I guess) complain as soon as I try to do something afterwards, even only exiting the program :

Traceback (most recent call last):
  File "test_thread.py", line 27, in <module>
    show()
  File "/usr/lib/pymodules/python2.7/matplotlib/pyplot.py", line 139, in show
    _show(*args, **kw)
  File "/usr/lib/pymodules/python2.7/matplotlib/backend_bases.py", line 83, in __call__
    manager.show()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 444, in show
    self.canvas.draw_idle()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 258, in draw_idle
    self._idle_callback = self._tkcanvas.after_idle(idle_draw)
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 512, in after_idle
    return self.after('idle', func, *args)
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 504, in after
    name = self._register(callit)
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1101, in _register
    self.tk.createcommand(name, f)
RuntimeError: main thread is not in main loop
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/_pylab_helpers.py", line 82, in destroy_all
    manager.destroy()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 452, in destroy
    self.canvas._tkcanvas.after_cancel(self.canvas._idle_callback)
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 519, in after_cancel
    data = self.tk.call('after', 'info', id)
RuntimeError: main thread is not in main loop
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/_pylab_helpers.py", line 82, in destroy_all
    manager.destroy()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 452, in destroy
    self.canvas._tkcanvas.after_cancel(self.canvas._idle_callback)
  File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 519, in after_cancel
    data = self.tk.call('after', 'info', id)
RuntimeError: main thread is not in main loop

我的第一个想法是在每个fork()上都复制了TK上下文，这在某种程度上干扰了主过程中的TK循环，但是我没有在我的工作人员中做任何与TK相关的事情.有什么想法吗?

My first thought was that the TK context was duplicated on each fork(), which somehow interferred with the TK loop in the main process, but I'm not doing anything TK-related in my workers. Any ideas?

推荐答案

错误消息参考Tkinter.因此，看起来您正在使用TkAgg后端.后面的代码是特定于TkAgg/Tkinter的.特别是通话

The error messages reference Tkinter. So it looks like you are using the TkAgg backend. The code that follows is TkAgg/Tkinter-specific. In particular the call

win.after(100, animate)

使用特定于Tkinter的after方法. GtkAgg/PyGtk有一个类似的调用，其他后端也有类似的调用.但是我只想强调一下，接下来是TkAgg/Tkinter特定的.

makes use of the Tkinter-specific after method. There is an analogous call for GtkAgg/PyGtk, and similarly for other backends. But I just want to stress that what follows is TkAgg/Tkinter-specific.

Tkinter旨在在单个线程中运行.也就是说，所有Tkinter GUI调用都应源自单个线程(通常，不一定是主线程).

Tkinter is intended to be run in a single thread. That is, all Tkinter GUI calls should originate from a single thread (usually, be not necessarily, the main thread).

池的apply_async回调方法在主进程中的单独线程(_handle_results)中运行.由于imshow()是从Pool的_handle_results线程中调用的，而show()是在主线程中调用的，因此Tkinter会抱怨

The Pool's apply_async callback method runs in a separate (_handle_results) thread in the main process. Since imshow() is called from the Pool's _handle_results thread and show() is called in the main thread, Tkinter complains

RuntimeError: main thread is not in main loop

在这种情况下，我看不到使用apply_async回调的方法.

I don't see a way to use the apply_async callback in this situation.

相反，我们可以做的是安排do_work将out_image放入multiprocessing.Queue()(在下面的代码中称为out_queue).然后，我们将让主进程的主线程在此队列中轮询项目，并在它们从队列中出来时显示它们.此轮询是在下面的animate函数中完成的.

Instead, what we can do is arrange for do_work to put out_image in a multiprocessing.Queue() (which I call out_queue in the code below). We'll then have the main process's main thread poll this Queue for items and display them as they come out of the Queue. This polling is done in the animate function, below.

plt.ion()仅用于交互式会话.尽管有时可以编写一些小的脚本，这些脚本似乎可以与plt.ion()一起使用，但如果您拒绝在脚本中使用plt.ion()，而是编写尊重GUI框架事件循环的代码，则可以得到更好的结果和更干净的GUI

plt.ion() is meant for interactive sessions only. Although it is sometimes possible to write little scripts which sort-of seem to work with plt.ion(), you'll get better results and cleaner GUIs if you resist using plt.ion() in scripts and instead write code that respects the GUI framework's event loop.

尽管可能可以修复脚本并使用plt.ion()，但由于这不是编写matplotlib脚本的推荐方法，所以让我们看看是否可以避免这样做.

Although it is probably possible to fix your script and use plt.ion(), since this is not the recommended way of writing matplotlib scripts, let's see if we can avoid doing that.

plt.show()告诉Tkinter运行其事件循环.请注意，进行此调用后，将绘制GUI窗口，您可以单击按钮，放大和缩小等.

plt.show() tells Tkinter to run its event loop. Notice that once this call is made, the GUI window is drawn, you can click buttons, zoom in and out, etc.

以某种方式，我们需要将一个函数注入到此事件循环中，以由该事件循环定期运行，并与可能发生的所有其他GUI事件协作.我们希望该函数检查是否有任何辅助子进程为我们提供输出，如果有，则更新imshow图像.

Somehow we need to inject a function into this event loop, to be run periodically by the event loop, and cooperatively with all the other GUI events that might be occurring. We want this function to check if any of our worker subprocesses have output for us, and if so, to update the imshow image.

对于TkAgg/Tkinter，注入这种功能的方法是

With TkAgg/Tkinter, the way to inject such a function is

win = fig.canvas.manager.window
win.after(100, animate)

这将告诉Tkinter在大约100ms之后运行一次animate函数.由于我们希望功能animate定期运行，因此我们只需要粘贴另一个

This will tell Tkinter to run the function animate (once) after (about) 100ms have elapsed. Since we want the function animate to run periodically, we just stick another

win.after(100, animate)

在animate末尾致电.

import matplotlib as mpl
mpl.use('TkAgg')
import matplotlib.pyplot as plt
import numpy as np
import multiprocessing as mp
import logging
import Queue
logger = mp.log_to_stderr(logging.INFO)

# Only does something to in_image, doesn't access anything else
def do_work(in_image):
    logger.info('Processing in_image')
    for x in xrange(100000):
        out_image = in_image[::-1, ::-1]
    out_queue.put(out_image)

# Update the output image and display if needed
out_all = np.zeros((256, 256))


def pool_initializer(out_queue_):
    # Setup out_queue as a global variable *in the worker subprocesses*
    global out_queue
    out_queue = out_queue_


def animate():
    global out_all
    try:
        out_image = out_queue.get_nowait()
    except Queue.Empty:
        pass
    else:
        logger.info("Updating")
        out_all += out_image
        im.set_data(out_all)
        fig.canvas.draw()  # redraw the canvas
    win.after(100, animate)

if __name__ == '__main__':
    out_queue = mp.Queue()
    logger.info("Starting pool")
    pool = mp.Pool(initializer=pool_initializer, initargs=(out_queue, ))
    work = [np.random.random((256, 256)) for f in range(20)]
    for o in work:
        pool.apply_async(do_work, [o])
    pool.close()

    fig, ax = plt.subplots()
    win = fig.canvas.manager.window
    # Output image
    im = plt.imshow(out_all, vmin=0, vmax=1)

    # Register a function to be run once
    win.after(100, animate)
    plt.show()
    logger.info("Done")

这篇关于Matplotlib和多处理RuntimeError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！