如何解决 python 多处理 matplotlib savefig() 问题? [英] How to fix the python multiprocessing matplotlib savefig() issue?

问题描述

我想通过多处理模块为许多图形加速 matplotlib.savefig()，并尝试对并行和序列之间的性能进行基准测试.

I want to speed up matplotlib.savefig() for many figures by multiprocessing module, and trying to benchmark the performance between parallel and sequence.

以下是代码:

# -*- coding: utf-8 -*-
"""
Compare the time of matplotlib savefig() in parallel and sequence
"""

import numpy as np
import matplotlib.pyplot as plt
import multiprocessing
import time


def gen_fig_list(n):
    ''' generate a list to contain n demo scatter figure object '''
    plt.ioff()
    fig_list = []
    for i in range(n):
        plt.figure();
        dt = np.random.randn(5, 4);
        fig = plt.scatter(dt[:,0], dt[:,1], s=abs(dt[:,2]*1000), c=abs(dt[:,3]*100)).get_figure()
        fig.FM_figname = "img"+str(i)
        fig_list.append(fig)
    plt.ion()
    return fig_list


def savefig_worker(fig, img_type, folder):
    file_name = folder+"\"+fig.FM_figname+"."+img_type
    fig.savefig(file_name, format=img_type, dpi=fig.dpi)
    return file_name


def parallel_savefig(fig_list, folder):
    proclist = []
    for fig in fig_list:
        print fig.FM_figname,
        p = multiprocessing.Process(target=savefig_worker, args=(fig, 'png', folder)) # cause error
        proclist.append(p)
        p.start()

    for i in proclist:
        i.join()



if __name__ == '__main__':
    folder_1, folder_2 = 'Z:\A1', 'Z:\A2'
    fig_list = gen_fig_list(10)

    t1 = time.time()
    parallel_savefig(fig_list,folder_1)
    t2 = time.time()
    print '
Mulprocessing time    : %0.3f'%((t2-t1))

    t3 = time.time()
    for fig in fig_list:
        savefig_worker(fig, 'png', folder_2)
    t4 = time.time()
    print 'Non_Mulprocessing time: %0.3f'%((t4-t3))

我遇到了问题此应用程序已请求运行时以不寻常的方式终止它.请联系应用程序的支持团队以获取更多信息." 由 p = 多处理引起的错误.Process(target=savefig_worker, args=(fig, 'png', folder)) .

And I meet problem "This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information." error caused by p = multiprocessing.Process(target=savefig_worker, args=(fig, 'png', folder)) .

为什么?又该如何解决呢?

Why ? And how to solve it ?

(Windows XP + Python:2.6.1 + Numpy:1.6.2 + Matplotlib:1.2.0)

(Windows XP + Python: 2.6.1 + Numpy: 1.6.2 + Matplotlib: 1.2.0)

(在 python 2.7.3 上添加错误消息)

在 python 2.7.3 的 IDLE 上运行时，它给出以下错误消息:

When run on IDLE of python 2.7.3, it gives below error msg:

>>> 
img0

Traceback (most recent call last):
  File "C:Documents and SettingsAdministratordesktopmulsavefig_pilot.py", line 61, in <module>
    proc.start()
  File "d:Python27libmultiprocessingprocess.py", line 130, in start

  File "d:Python27libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "d:Python27libpickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <function notify_axes_change at 0x029F5030>: it's not found as matplotlib.backends.backend_qt4.notify_axes_change

(我的解决方案演示)

灵感来自 Matplotlib:在多个线程中同时绘图

# -*- coding: utf-8 -*-
"""
Compare the time of matplotlib savefig() in parallel and sequence
"""

import numpy as np
import matplotlib.pyplot as plt
import multiprocessing
import time


def gen_data(fig_qty, bubble_qty):
    ''' generate data for fig drawing '''
    dt = np.random.randn(fig_qty, bubble_qty, 4)
    return dt


def parallel_savefig(draw_data, folder):
    ''' prepare data and pass to worker '''

    pool = multiprocessing.Pool()

    fig_qty = len(draw_data)
    fig_para = zip(range(fig_qty), draw_data, [folder]*fig_qty)

    pool.map(fig_draw_save_worker, fig_para)
    return None


def fig_draw_save_worker(args):
    seq, dt, folder = args
    plt.figure()
    fig = plt.scatter(dt[:,0], dt[:,1], s=abs(dt[:,2]*1000), c=abs(dt[:,3]*100), alpha=0.7).get_figure()
    plt.title('Plot of a scatter of %i' % seq)
    fig.savefig(folder+"\"+'fig_%02i.png' % seq)
    plt.close()
    return None


if __name__ == '__main__':
    folder_1, folder_2 = 'A1', 'A2'
    fig_qty, bubble_qty =  500, 100
    draw_data = gen_data(fig_qty, bubble_qty)

    print 'Mulprocessing  ...   ',
    t1 = time.time()
    parallel_savefig(draw_data, folder_1)
    t2 = time.time()
    print 'Time : %0.3f'%((t2-t1))

    print 'Non_Mulprocessing .. ', 
    t3 = time.time()
    for para in zip(range(fig_qty), draw_data, [folder_2]*fig_qty):
        fig_draw_save_worker(para)
    t4 = time.time()
    print 'Time : %0.3f'%((t4-t3))

    print 'Speed Up: %0.1fx'%(((t4-t3)/(t2-t1)))

推荐答案

这并不是一个真正的错误，也就是说，更多的是一个限制.

It is not really a bug, per-say, more of a limitation.

解释在你的错误信息的最后一行:

The explanation is in the last line of your error mesage:

PicklingError: Can't pickle <function notify_axes_change at 0x029F5030>: it's not found as matplotlib.backends.backend_qt4.notify_axes_change

它告诉你图形对象的元素不能被腌制，这就是MultiProcess在进程之间传递数据的方式.对象在主要过程中被腌制，作为腌制运输，然后在另一侧重新构建.即使您修复了 this 确切的问题(可能通过使用不同的后端，或剥离有问题的功能(这可能会以其他方式破坏事情))我很确定 的核心部分无法腌制的图形、Axes或Canvas对象.

It is telling you that elements of the figure objects can not be pickled, which is how MultiProcess passes data between the processes. The objects are pickled in the main processes, shipped as pickles, and then re-constructed on the other side. Even if you fixed this exact issue (maybe by using a different backend, or stripping off the offending function (which might break things in other ways)) I am pretty sure there are core parts of Figure, Axes, or Canvas objects that can not be pickled.

正如@bigbug 所指出的，如何解决此限制的示例，Matplotlib:在多个线程中同时绘图.基本思想是您将 整个 绘图例程推送到子流程，因此您只推送 numpy 数组和可能跨流程边界的一些配置信息.

As @bigbug point to, an example of how to get around this limitation, Matplotlib: simultaneous plotting in multiple threads. The basic idea is that you push your entire plotting routine off to the sub-process so you only push numpy arrays an maybe some configuration information across the process boundry.

这篇关于如何解决 python 多处理 matplotlib savefig() 问题?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！