为什么Python对于一个简单的for循环这么慢? [英] Why Python is so slow for a simple for loop?

问题描述

我们正在用Python实现一些kNN和SVD实现.其他人选择了Java.我们的执行时间非常不同.我使用cProfile来查看哪里出错了，但实际上一切都很很好.是的，我也使用numpy.但是我想问一个简单的问题.

We are making some kNN and SVD implementations in Python. Others picked Java. Our execution times are very different. I used cProfile to see where I make mistakes but everything is quite fine actually. Yes, I use numpy also. But I would like to ask simple question.

total = 0.0
for i in range(9999): # xrange is slower according 
    for j in range(1, 9999):            #to my test but more memory-friendly.
        total += (i / j)
print total

此代码段在我的计算机上耗时31.40s.

This snippet takes 31.40s on my computer.

此代码的Java版本在同一台计算机上花费1秒或更短的时间.我想类型检查是此代码的主要问题.但是我应该为我的项目做很多这样的操作，我认为9999 * 9999的数字不是很大.

Java version of this code takes 1 second or less on the same computer. Type checking is a main problem for this code, I suppose. But I should make lots of operation like this for my project and I think 9999*9999 is not so big number.

我想我犯错了，因为我知道Python已被许多科学项目所使用.但是为什么这段代码这么慢，我该如何处理更大的问题呢?

I think I am making mistakes because I know Python is used by lots of scientific projects. But why is this code so slow and how can I handle problems bigger than this?

我应该使用诸如Psyco之类的JIT编译器吗?

Should I use a JIT compiler such as Psyco?

我也说这个循环问题只是一个例子.代码并不像这样简单，因此很难将改进/代码示例付诸实践.

I also say that this loop problem is only an example. The code is not as simple as like this and It may be tough to put into practice your improvements/code samples.

另一个问题是，我可以实施大量数据挖掘吗?如果正确使用numpy和scipy的机器学习算法?

Another question is that can I implement lots of data mining & machine learning algorithms with numpy and scipy if I use it correctly?

推荐答案

我想我犯了错​​误，因为我知道Python已被许多科学项目使用.

I think I am making mistakes because I know Python is used by lots of scientific projects.

他们大量使用SciPy(NumPy是最重要的组件，但我听说围绕NumPy API开发的生态系统更为重要)，大大可加快这些项目的各种操作需要.这是您做错的事情:您不是在用C编写 critical 代码.Python通常适合于开发，但是放置适当的扩展模块本身就是一项至关重要的优化(至少当您处理数字时). Python是一种非常糟糕的语言，可以在其中实现紧密的内部循环.

They're heavily using SciPy (NumPy being the most prominent component, but I've heard the ecosystem that developed around NumPy's API is even more important) which vastly speeds up all kinds operations these projects need. There's what you are doing wrong: You aren't writing your critical code in C. Python is great for developing in general, but well-placed extension modules are a vital optimization in its own right (at least when you're crunching numbers). Python is a really crappy language to implement tight inner loops in.

默认的(目前是最受欢迎和广泛支持的)实现是一个简单的字节码解释器.即使是最简单的操作(例如整数除法)，也可能需要数百个CPU周期，多个内存访问(常见的类型检查)，多个C函数调用等，而不是几个(对于整数，甚至是单个)除法)指令.此外，该语言具有许多抽象设计，这些抽象增加了开销.如果使用xrange，循环将在堆上分配9999个对象；如果使用range，则将在堆上分配更多的对象(9999 * 9999整数减去256 * 256左右的高速缓存小整数).此外，xrange版本在每次迭代时都调用一种方法来进行扩展-如果未专门优化序列迭代，则range版本也是如此.尽管如此，它仍然需要整个字节码调度，这本身是非常复杂的(当然，与整数除法相比).

The default (and for the time being most popular and widely-supported) implementation is a simple bytecode interpreter. Even the simplest operations, like an integer division, can take hundreds of CPU cycles, multiple memory accesses (type checks being a popular example), several C function calls, etc. instead of a few (or even single, in the case of integer division) instruction. Moreover, the language is designed with many abstractions which add overhead. Your loop allocates 9999 objects on the heap if you use xrange - far more if you use range (9999*9999 integer minus around 256*256 for small integers which are cached). Also, the xrange version calls a method on each iteration to advance - the range version would too if iteration over sequences hadn't been optimized specifically. It still takes a whole bytecode dispatch though, which is itself vastly complex (compared to an integer division, of course).

看看什么是JIT很有趣(我建议使用PyPy而不是Psyco，后者不再积极开发，而且范围也很有限-尽管对于这个简单的示例来说，它可能很好用).经过一小部分的迭代后，它应该会产生一个几乎最佳的机器代码循环，并增加一些保护措施-简单的整数比较，如果失败，则跳转-以保持正确性，以防您在该列表中出现字符串. Java可以做同样的事情，只是更快(不必先跟踪)，并且只需较少的防护措施(至少在使用int的情况下).这就是为什么它要快得多的原因.

It would be interesting to see what a JIT (I'd recommend PyPy over Psyco, the latter isn't actively developed anymore and very limited in scope anyway - it might work well for this simple example though). After a tiny fraction of iterations, it should produce a nigh-optimal machine code loop augmented with a few guards - simple integer comparisions, jumping if they fail - to maintain correctness in case you got a string in that list. Java can do the same thing, only sooner (it doesn't have to trace first) and with fewer guards (at least if you use ints). That's why it's so much faster.

这篇关于为什么Python对于一个简单的for循环这么慢?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！