Python垃圾收集可能会很慢？ [英] Python garbage collection can be that slow?

问题描述

我的python应用程序有问题，我认为它与python垃圾回收有关，即使我不确定...

问题是我的应用程序需要大量的时间来退出并切换到下一个功能。

在我的应用程序中，我处理非常大的字典，包含数千个大从包装的C ++类实例化的对象。

我在程序中放置了一些时间戳输出，并且在每个函数的末尾看到函数内部创建的对象应该超出范围，在调用下一个函数之前，解释器花费了大量的时间。我在程序结束时观察到了同样的问题，程序应该退出：在屏幕上的最后一个时间戳和新提示的出现之间花费了很多时间（〜小时！）。

内存使用情况稳定，所以我没有真正的内存泄露。

有什么建议？

可以成千上万个大型C ++对象的垃圾回收缓慢吗？

有没有一种方法可以加速这个速度？

$ b $ p 更新：

非常感谢您的答案，您给了我很多提示调试我的代码：）

我在Scientific Linux 5上使用Python 2.6.5，这是一个基于红帽企业版5的定制发行版。
实际上，我没有使用SWIG来获取我们的C ++代码的Python绑定，而是使用Reflex / PyROOT框架。我知道，它在粒子物理学之外还不是很清楚（但仍然是开放源代码并免费提供），我必须使用它，因为它是我们主框架的默认设置。

和在这种情况下，来自Python端的DEL命令不起作用，我已经尝试过了。 DEL只会删除链接到C ++对象的python变量，而不是内存中的对象本身，它仍然由C ++端拥有...

...我知道，它不是标准我猜，有点复杂，对不起：-P

但是按照你的提示，我会分析我的代码，然后我会回到你有更多的细节，正如你所建议的。

附加更新：

，根据你的建议，我用 cProfile 来检测我的代码，并且发现实际上 gc.collect（）函数这个函数占用了大部分的运行时间!!

这里输出来自 cProfile + pstats print_stats（）：

 
 
 >>> p.sort_stats（time ）.print_stats（20）
 Wed 10 20 17:46:02 2010 mainProgram.profile 
 
 547303 548.060 CPU秒内的函数调用（542629原始调用）
 
订购人：内部时间
列表由于限制而从727减少到20 
 
 ncalls tottime percall cumtime percall文件名：lineno（功能）
 4 345.701 86.425 345.704 86.426 {gc.collect} 
 1 167.115 167.115 200.946 200.946 PlotD3PD_v3.2.py:2041(PlotSamplesBranches）
 28 12.817 0.458 13.345 0.477 PlotROOTUtils.py:205（SaveItems）
 9900 10.425 0.001 10.426 0.001 PlotD3PD_v3.2.py:1973(HistoStyle）
 6622 5.188 0.001 5.278 0.001 PlotROOTUtils.py:403(__init__）
 57 0.625 0.011 0.625 0.011 {内置方法负载} 
 103 0.625 0.006 0.792 0.008 dbutils.py:41 (DeadlockWrap）
 14 0.475 0.034 0.475 0.034 {cPickle.Pickler'对象的方法'转储'} 
 6622 0.453 0.000 5.908 0.001 PlotROOTUtils.py:421（创建可用图形）
 26455 0.434 0.000 0.508 0.000 / opt / root / lib / ROOT.py：215（__ getattr__）
 [...] 
 
 >>> p.sort_stats（cumulative）。print_stats（20）
 Wed Oct 20 17:46:02 2010 mainProgram.profile 
 
 547303 548.060函数调用（542629原语调用） CPU秒
 
按顺序：累计时间
由于限制，列表从727减少到20 
 $ b ncalls tottime percall cumtime percall文件名：lineno（功能）
 1 0.001 0.001 548.068 548.068 PlotD3PD_v3.2.py:2492（主）
 4 0.000 0.000 346.756 86.689 /usr/lib//lib/python2.5/site-packages/guppy/heapy/Use.py:171 （堆）
 4 0.005 0.001 346.752 86.688 /usr/lib//lib/python2.5/site-packages/guppy/heapy/View.py:344(heap）
 1 0.002 0.002 346.147 346.147 PlotD3PD_v3 .2.py：2537（LogAndFinalize）
 4 345.701 86.425 345.704 86.426 {gc.collect} 
 1 167.115 167.115 200.946 200.946 PlotD3PD_v3.2.py:2041(PlotBranches）
 28 12.817 0.458 13.345 0.477 PlotROOTUtils.py:205（S aveItems）
 9900 10.425 0.001 10.426 0.001 PlotD3PD_v3.2.py:1973(HistoStyle）
 13202 0.336 0.000 6.818 0.001 PlotROOTUtils.py:431(PlottingCanvases）
 6622 0.453 0.000 5.908 0.001 / root / svn_co / rbianchi / SoftwareDevelopment 
 
 [...] 
 
 >>> 
 

因此，在两个输出中，分别按时间和累计时间排序， gc.collect（）是消耗我大部分运行时间的函数程序！ ：-P

这是内存分析器 Heapy 的输出，在返回<$ c $

 
返回前的内存使用情况：
一组分区65901个对象。总大小= 4765572字节。 
索引计数％大小％累计％种类（班级/字典）
 0 25437 39 1452444 30 1452444 30 str 
 1 6622 10 900592 19 2353036 49字典PlotROOTUtils.Canvas 
 2 109 0 567016 12 2920052 61模块字典
 3 7312 11 280644 6 3200696 67元组
 4 6622 10 238392 5 3439088 72 0xa4ab74c 
 5 6622 10 185416 4 3624504 76 PlotROOTUtils.Canvas 
 6 2024 3 137632 3 3762136 79 types.CodeType 
 7 263 0 129080 3 3891216 82 dict（无所有者）
 8 254 0 119024 2 4010240 84类型
的字典9 254 0 109728 2 4119968 86类型
指数数量％大小％累积百分比种类（班级类别）
 10 1917 3 107352 2 4264012 88功能
 11 3647 5 102116 2 4366128 90 ROOT.MethodProxy 
 12 148 0 80800 2 4446928 92类字典
 13 110 9 2 39924 1 4486852 93 __builtin __。wrapper_descriptor 
 14 239 0 23136 0 4509988 93 list 
 15 87 0 22968 0 4532956 94字典guppy.etc.Glue.Interface 
 16 644 1 20608 0 4553564 94 types.BuiltinFunctionType 
 17 495 1 19800 0 4573364 94 __builtin __。weakref 
 18 23 0 11960 0 4585324 95字典guppy.etc.Glue.Share 
 19 367 1 11744 0 4597068 95 __builtin __。method_descriptor 
 
 

任何想法为什么或者如何优化垃圾回收？

有没有更详细的检查我可以做？

解决方案

这是Python 2.6中的已知垃圾回收器问题，当分配许多对象而不释放其中的任何对象时，导致垃圾收集的二次时间。大量列表的人口。

有两个简单的解决方案：

1. 之后列出并启用它。

   
   

     l = [] 
    gc.disable（）
    for x in xrange（10 ** 6）：
    l.append（x）
    gc.enable（）
     

   
   




2. 或更新至 Python 2.7，问题已解决

我更喜欢第二种解决方案，但它并不总是一个选项;）

I have a problem with my python application, and I think it's related to the python garbage collection, even if I'm not sure...

The problem is that my application takes a lot of time to exit and to switch to one function to the next one.

In my application I handle very large dictionaries, containing thousands of large objects which are instantiated from wrapped C++ classes.

I put some timestamp outputs in my program, and I saw that at the end of each function, when objects created inside the function should go out of scope, a lot of time is spent by the interpreter before calling the next function. And I observe the same problem at the end of the application, when the program should exit: a lot of time (~ hours!) is spent between the last timestamp on screen and the appearance of the fresh prompt.

The memory usage is stable, so I don't really have memory leaks.

Any suggestions?

Can be the garbage collection of thousands of large C++ objects that slow?

Is there a method to speed up that?

UPDATE:

Thanks a lot for all your answers, you gave me a lot of hints to debug my code :-)

I use Python 2.6.5 on Scientific Linux 5, a customized distribution based on Red Hat Enterprise 5. And actually I'm not using SWIG to get Python bindings for our C++ code, but the Reflex/PyROOT framework. I know, it's not very known outside particle physics (but still open source and freely available) and I have to use it because it's the default for our main framework.

And in this context the DEL command from the Python side does not work, I had already tried it. DEL only deletes the python variable linked to the C++ object, not the object itself in memory, which is still owned by the C++ side...

...I know, it's not-standard I guess, and a bit complicated, sorry :-P

But following your hints, I'll profile my code and I'll come back to you with more details, as you suggested.

ADDITIONAL UPDATE:

Ok, following your suggestions, I instrumented my code with cProfile, and I discovered that actually the gc.collect() function is the function taking the most of the running time!!

Here the output from cProfile + pstats print_stats():

    >>> p.sort_stats("time").print_stats(20)
Wed Oct 20 17:46:02 2010    mainProgram.profile

         547303 function calls (542629 primitive calls) in 548.060 CPU seconds

   Ordered by: internal time
   List reduced from 727 to 20 due to restriction 

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        4  345.701   86.425  345.704   86.426 {gc.collect}
        1  167.115  167.115  200.946  200.946 PlotD3PD_v3.2.py:2041(PlotSamplesBranches)
       28   12.817    0.458   13.345    0.477 PlotROOTUtils.py:205(SaveItems)
     9900   10.425    0.001   10.426    0.001 PlotD3PD_v3.2.py:1973(HistoStyle)
     6622    5.188    0.001    5.278    0.001 PlotROOTUtils.py:403(__init__)
       57    0.625    0.011    0.625    0.011 {built-in method load}
      103    0.625    0.006    0.792    0.008 dbutils.py:41(DeadlockWrap)
       14    0.475    0.034    0.475    0.034 {method 'dump' of 'cPickle.Pickler' objects}
     6622    0.453    0.000    5.908    0.001 PlotROOTUtils.py:421(CreateCanvas)
    26455    0.434    0.000    0.508    0.000 /opt/root/lib/ROOT.py:215(__getattr__)
[...]

>>> p.sort_stats("cumulative").print_stats(20)
Wed Oct 20 17:46:02 2010    mainProgram.profile

         547303 function calls (542629 primitive calls) in 548.060 CPU seconds

   Ordered by: cumulative time
   List reduced from 727 to 20 due to restriction 

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.001    0.001  548.068  548.068 PlotD3PD_v3.2.py:2492(main)
        4    0.000    0.000  346.756   86.689 /usr/lib//lib/python2.5/site-packages/guppy/heapy/Use.py:171(heap)
        4    0.005    0.001  346.752   86.688 /usr/lib//lib/python2.5/site-packages/guppy/heapy/View.py:344(heap)
        1    0.002    0.002  346.147  346.147 PlotD3PD_v3.2.py:2537(LogAndFinalize)
        4  345.701   86.425  345.704   86.426 {gc.collect}
        1  167.115  167.115  200.946  200.946 PlotD3PD_v3.2.py:2041(PlotBranches)
       28   12.817    0.458   13.345    0.477 PlotROOTUtils.py:205(SaveItems)
     9900   10.425    0.001   10.426    0.001 PlotD3PD_v3.2.py:1973(HistoStyle)
    13202    0.336    0.000    6.818    0.001 PlotROOTUtils.py:431(PlottingCanvases)
     6622    0.453    0.000    5.908    0.001 /root/svn_co/rbianchi/SoftwareDevelopment

[...]

>>>

So, in both outputs, sorted by "time" and by "cumulative" time respectively, gc.collect() is the function consuming the most of the running time of my program! :-P

And this is the output of the memory profiler Heapy, just before returning the main() program.

memory usage before return:
Partition of a set of 65901 objects. Total size = 4765572 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0  25437  39  1452444  30   1452444  30 str
     1   6622  10   900592  19   2353036  49 dict of PlotROOTUtils.Canvas
     2    109   0   567016  12   2920052  61 dict of module
     3   7312  11   280644   6   3200696  67 tuple
     4   6622  10   238392   5   3439088  72 0xa4ab74c
     5   6622  10   185416   4   3624504  76 PlotROOTUtils.Canvas
     6   2024   3   137632   3   3762136  79 types.CodeType
     7    263   0   129080   3   3891216  82 dict (no owner)
     8    254   0   119024   2   4010240  84 dict of type
     9    254   0   109728   2   4119968  86 type
  Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
    10   1917   3   107352   2   4264012  88 function
    11   3647   5   102116   2   4366128  90 ROOT.MethodProxy
    12    148   0    80800   2   4446928  92 dict of class
    13   1109   2    39924   1   4486852  93 __builtin__.wrapper_descriptor
    14    239   0    23136   0   4509988  93 list
    15     87   0    22968   0   4532956  94 dict of guppy.etc.Glue.Interface
    16    644   1    20608   0   4553564  94 types.BuiltinFunctionType
    17    495   1    19800   0   4573364  94 __builtin__.weakref
    18     23   0    11960   0   4585324  95 dict of guppy.etc.Glue.Share
    19    367   1    11744   0   4597068  95 __builtin__.method_descriptor

Any idea why, or how to optimize the garbage collection?

Is there any more detailed check I can do?

解决方案

This is known garbage collector issue in Python 2.6 causing quadratic time for garbage collection when many objects are being allocated without deallocating any of them ie. population of large list.
There are two simple solutions:

1. either disable garbage collection before populating large lists and enable it afterwards

   l = []
   gc.disable()
   for x in xrange(10**6):
     l.append(x)
   gc.enable()
   

2. or update to Python 2.7, where the issue has been solved

I prefer the second solution, but it's not always an option;)

这篇关于Python垃圾收集可能会很慢？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！