存在大量列表时，Python函数会变慢 [英] Python function slows down with presence of large list

问题描述

我正在测试几种不同方法对某些数据进行复杂迭代的速度，但发现有些奇怪.似乎在某个功能的本地存在一个较大的列表会大大降低该功能的速度，即使该功能未触及该列表也是如此.例如，通过同一生成器函数的2个实例创建2个独立列表的速度第二次慢约2.5倍.如果在创建第二个列表之前删除了第一个列表，则两个迭代器的速度相同.

I was testing the speeds of a few different ways to do complex iterations over some of my data, and I found something weird. It seems that having a large list local to some function slows down that function considerably, even if it is not touching that list. For example, creating 2 independent lists via 2 instances of the same generator function is about 2.5x slower the second time. If the first list is removed prior to creating the second, both iterators go at the same spee.

def f():  
    l1, l2 = [], []  
    for c1, c2 in generatorFxn():  
        l1.append((c1, c2))  
    # destroying l1 here fixes the problem 
    for c3, c4 in generatorFxn():  
        l2.append((c3, c4))

每个列表最终约有310万个条目长，但是我看到较小的列表也有相同的效果.第一个for循环大约需要4.5秒，第二个循环需要10.5.如果在注释位置插入l1= []或l1= len(l1)，则两个for循环都需要4.5秒.

The lists end up about 3.1 million items long each, but I saw the same effect with smaller lists too. The first for loop takes about 4.5 seconds to run, the second takes 10.5. If I insert l1= [] or l1= len(l1) at the comment position, both for loops take 4.5 seconds.

为什么函数中的本地内存分配速度与该函数的变量的当前大小有关系?

Why does the speed of local memory allocation in a function have anything to do with the current size of that function's variables?

禁用垃圾收集器会修复所有问题，因此必须归因于其不断运行.结案了！

Disabling the garbage collector fixes everything, so must be due to it running constantly. Case closed!

推荐答案

当创建那么多新对象(300万个元组)时，垃圾收集器陷入了泥潭.如果您使用gc.disable()关闭垃圾收集，问题就消失了(程序的启动速度提高了4倍).

When you create that many new objects (3 million tuples), the garbage collector gets bogged down. If you turn off garbage collection with gc.disable(), the issue goes away (and the program runs 4x faster to boot).

这篇关于存在大量列表时，Python函数会变慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！