如何在Python中设置字典的初始大小？ [英] How to set initial size for a dictionary in Python?

问题描述

我把约400万个不同的键放入Python字典。
创建此字典大约需要15分钟，并在我的机器上消耗大约4GB的内存。字典完全创建后，查询字典很快。

我怀疑字典的创建是如此耗费资源，因为字典经常被复制（因为它增长很大） 。
是否可以在Python中创建一个具有初始大小或存储区号的字典？

我的字典从一个数字到一个对象。

  class MyObject（object）：
 def __init __（self）：
＃some fields ... 
 
d = {} 
d [i] = MyObject（）＃4M次在不同的键... 
  

解决方案

在性能问题上，总是最好的测量。以下是一些时间：

  d = {} 
在我的xrange（4000000）中：
d [ i] =无
＃722ms 
 
d = dict（itertools.izip（xrange（4000000），itertools.repeat（无）））
＃634ms 
 
 dict.fromkeys（xrange（4000000））
＃558ms 
 
s = set（xrange（4000000））
 dict.fromkeys（s）
＃不包括设置构造353ms 
  

最后一个选项不会进行任何调整大小，它只是从集合中复制散列并增加引用。正如你所看到的，调整大小并不花费大量的时间。这可能是您的对象创建速度很慢。

I'm putting around 4 million different keys into a Python dictionary. Creating this dictionary takes about 15 minutes and consumes about 4GB of memory on my machine. After the dictionary is fully created, querying the dictionary is fast.

I suspect that dictionary creation is so resource consuming as the dictionary is very often rehashed (as it grows enormously). Is is possible to create a dictionary in Python with some initial size or bucket number?

My dictionary points from a number to an object.

class MyObject(object):
  def __init__(self):
    # some fields...

d = {}
d[i] = MyObject()  # 4M times on different key...

解决方案

With performance issues it's always best to measure. Here are some timings:

 d = {}
 for i in xrange(4000000):
     d[i] = None
 # 722ms

 d = dict(itertools.izip(xrange(4000000), itertools.repeat(None)))
 # 634ms

 dict.fromkeys(xrange(4000000))
 # 558ms

 s = set(xrange(4000000))
 dict.fromkeys(s)
 # Not including set construction 353ms

The last option doesn't do any resizing, it just copies the hashes from the set and increments references. As you can see, the resizing isn't taking a lot of time. It's probably your object creation that is slow.

这篇关于如何在Python中设置字典的初始大小？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！