zip（）函数问题 [英] zip() function troubles
问题描述
大家好，
我一直在调试一段
代码大幅减速的原因......事实证明这是拉链功能。在过去
，压缩的列表相当短，但一旦大小
超过1000万，zip功能就会慢慢爬行。请注意，
有可用于存储超过1亿件物品的内存。
现在我知道zip（）浪费了大量内存，因为它复制了
列表的内容，我曾使用zip来尝试换取内存以获得速度
（嘿！），现在所有内容都被izip取代了它的工作原理只是
罚款。真正令人惊讶的是它在没有任何问题的情况下工作直到100万件物品，但是对于1000万件而言，它几乎可以说是坚实的。有谁知道为什么？它是否有一些限制，或者是否有一些关于操作系统的东西（在这种情况下为Vista）
使它表现得如此？
在尝试创建非常好的b / b $ b长列表时，我注意到了同样的行为，这些列表应该很容易适应内存，但高于给定的
门槛我得到了莫名的减速。现在我想到的是这个
关于扩展它们时列表增长的方式吗？
这里是代码：
来自itertools导入izip
BIGNUM = int（1E7）
＃let'列出一个大清单
data = range（BIGNUM）
＃这个工作正常（使用大约200 MB和4秒）
s = 0
for x in data：
s + = x
print s
＃这个工作正常，4秒以及
s = 0
表示x1，x2表示izip（数据，数据）：
s + = x1
print s
＃这需要2分钟！并使用600 MB的内存
＃内存使用量慢慢上升
s = 0
为x1，x2为zip（数据，数据） ：
s + = x1
print s
Istvan Albert< ;是*********** @ gmail.comwrites：
超过1000万zip函数减速到爬行。请注意，
有可用于存储超过1亿件物品的内存。
这是多少字节？也许这些项目（代码示例中的堆分配盒装
整数）比你预期的要大。
7月26日，7：44下午，Paul Rubin< http://phr...@NOSPAM.invalidwrote：
Istvan Albert< istvan.alb ... @ gmail.comwrites：
超过1000万zip函数减速到爬行。请注意，
有可用于存储超过1亿件物品的内存。
这是多少字节？也许这些项目（代码示例中的堆分配盒装
整数）比您预期的要大。
虽然我没有回答这个问题
我试图做的是我我相当肯定它是b / b
不是内存问题（某种交换），因为包含操作系统的整体内存使用量约为1Gb（超出可用范围） 2Gb）
我在带有4Gb内存的Linux服务器系统上进行了测试
a = [0] * 10 ** 7
需要几毫秒，但是说
b = zip（a，a）
需要一个非常很长时间才能完成：
atlas：〜
time python -c" a = [0] * 10 ** 7"
真实0m0.165s
用户0m0.128s
sys 0m0.036s
图册：〜
Hello all,
I''ve been debugging the reason for a major slowdown in a piece of
code ... and it turns out that it was the zip function. In the past
the lists that were zipped were reasonably short, but once the size
exceeded 10 million the zip function slowed to a crawl. Note that
there was memory available to store over 100 million items.
Now I know that zip () wastes lots of memory because it copies the
content of the lists, I had used zip to try to trade memory for speed
(heh!) , and now that everything was replaced with izip it works just
fine. What was really surprising is that it works with no issues up
until 1 million items, but for say 10 million it pretty much goes
nuts. Does anyone know why? is there some limit that it reaches, or is
there something about the operating system (Vista in the case) that
makes it behave like so?
I''ve noticed the same kinds of behavior when trying to create very
long lists that should easily fit into memory, yet above a given
threshold I get inexplicable slowdowns. Now that I think about is this
something about the way lists grow when expanding them?
and here is the code:
from itertools import izip
BIGNUM = int(1E7)
# let''s make a large list
data = range(BIGNUM)
# this works fine (uses about 200 MB and 4 seconds)
s = 0
for x in data:
s += x
print s
# this works fine, 4 seconds as well
s = 0
for x1, x2 in izip(data, data):
s += x1
print s
# this takes over 2 minutes! and uses 600 MB of memory
# the memory usage slowly ticks upwards
s = 0
for x1, x2 in zip(data, data):
s += x1
print s
Istvan Albert <is***********@gmail.comwrites:exceeded 10 million the zip function slowed to a crawl. Note that
there was memory available to store over 100 million items.How many bytes is that? Maybe the items (heap-allocated boxed
integers in your code example) are bigger than you expect.
On Jul 26, 7:44 pm, Paul Rubin <http://phr...@NOSPAM.invalidwrote:Istvan Albert <istvan.alb...@gmail.comwrites:exceeded 10 million the zip function slowed to a crawl. Note that
there was memory available to store over 100 million items.
How many bytes is that? Maybe the items (heap-allocated boxed
integers in your code example) are bigger than you expect.while I don''t have an answer to this
the point that I was trying to make is that I''m fairly certain that it
is not a memory issue (some sort of swapping) because the overall
memory usage with the OS included is about 1Gb (out of available 2Gb)
I tested this on a linux server system with 4Gb of RAM
a = [ 0 ] * 10**7
takes miliseconds, but say the
b = zip(a,a)
will take a very long time to finish:
atlas:~
time python -c "a = [0] * 10**7"
real 0m0.165s
user 0m0.128s
sys 0m0.036s
atlas:~
这篇关于zip（）函数问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
