提高python代码的速度 [英] Increasing speed of python code

问题描述

我有一些包含许多类的 Python 代码.我使用 cProfile 发现运行程序的总时间为 68 秒.我发现名为 Buyers 的类中的以下函数大约需要 68 秒中的 60 秒.我必须运行该程序大约 100 次，因此任何速度的提高都会有所帮助.你能提出通过修改代码来提高速度的方法吗?如果您需要更多有用的信息，请告诉我.

def qtyDemanded(self, timePd, priceVector):'''返回时间段时间Pd内的需求量.此外，还更新客户和非客户列表.输入:timePd 和 priceVector输出: priceVector[-1] <的人数公用事业'''## 将客户计数初始化为零## 将 self.customers 和 self.nonCustomers 设置为空列表价格 = priceVector[-1]计数 = 0self.customers = []self.nonCustomers = []对于 self.people 的人:如果 person.utility >= 价格:人.客户 = 1self.customers.append(person)别的:人.客户 = 0self.nonCustomers.append(person)返回 len(self.customers)

self.people 是 person 对象的列表.每个 person 都有 customer 和 utility 作为它的属性.

编辑 - 回复已添加

-------------------------------------

非常感谢您的建议.这里是回应人们善意的一些问题和建议制成.我还没有全部尝试过，但会尝试其他的，稍后再写.

(1) @amber - 该函数被访问了 80,000 次.

(2) @gnibbler 和其他人 - self.people 是内存中的 Person 对象列表.未连接到数据库.

(3) @Hugh Bothwell

原始函数占用的cumtime - 60.8 s(访问80000次)

带有本地函数别名的新函数占用的cumtime 建议 - 56.4 s(访问 80000 次)

(4) @rotoglup 和 @Martin Thomas

我还没有尝试过你的解决方案.我需要检查其余代码以查看我使用 self.customers 的地方，然后才能进行不将客户附加到 self.customers 列表的更改.但我会试试这个然后回信.

(5) @TryPyPy - 感谢您提供检查代码的好意.

让我先阅读一下您提出的建议，看看这些建议是否可行.

编辑 2有人建议，由于我在 self.people 中标记客户和非客户，我应该尝试不创建 self.customers 和 self.noncustomers<的单独列表/code> 使用附加.相反，我应该遍历 self.people 以查找客户数量.我尝试了以下代码并对 f_w_append 和 f_wo_append 下的两个函数进行计时.我确实发现后者花费的时间更少，但仍然是前者花费的时间的 96%.也就是说，速度的增加非常小.

@TryPyPy - 以下代码足以检查瓶颈函数，以防您的报价仍然存在以使用其他编译器进行检查.

再次感谢所有回复的人.

导入numpy类人(对象):def __init__(self, util):self.utility = utilself.customer = 0班级人口(对象):def __init__(self, numpeople):self.people = []self.cus = []self.noncus = []numpy.random.seed(1)utils = numpy.random.uniform(0, 300, numpeople)对于 utils 中的你:每 = 人 (u)self.people.append(per)popn = 人口(300)def f_w_append():'''带附加功能'''P = 75cus = []noncus = []对于每个 popn.people:如果 per.utility >= P:每个客户 = 1cus.append(per)别的:每个客户 = 0noncus.append(per)返回 len(cus)def f_wo_append():'''无附加功能'''P = 75对于每个 popn.people:如果 per.utility >= P:每个客户 = 1别的:每个客户 = 0客户数 = 0对于每个 popn.people:如果每个客户 == 1:客户数量 += 1返回客户数

编辑 3:问题似乎是 numpy

这是对 John Machin 在下面所说的话的回应.下面是定义 Population 类的两种方法.我运行了下面的程序两次，每次创建 Population 类的方法.一种使用 numpy，一种不使用 numpy.没有 numpy 的时间与约翰在跑步中发现的时间相似.一个 numpy 需要更长的时间.我不清楚的是 popn 实例是在时间记录开始之前创建的(至少这是从代码中显示出来的).那么，为什么 numpy 版本需要更长的时间.而且，我认为 numpy 应该更有效率.无论如何，问题似乎出在 numpy 而不是 append，即使它确实减慢了一些速度.有人可以用下面的代码确认吗?谢谢.

import random # 而不是 numpy导入 numpy导入时间timer_func = time.time # 使用 Mac OS X 10.5.8类人(对象):def __init__(self, util):self.utility = utilself.customer = 0类人口(对象):def __init__(self, numpeople):随机种子(1)self.people = [Person(random.uniform(0, 300)) for i in xrange(numpeople)]self.cus = []self.noncus = []# 基于 Numpy# 类人口(对象):# def __init__(self, numpeople):# numpy.random.seed(1)# utils = numpy.random.uniform(0, 300, numpeople)# self.people = [Person(u) for u in utils]# self.cus = []# self.noncus = []def f_wo_append(popn):'''无附加功能'''P = 75对于每个 popn.people:如果 per.utility >= P:每个客户 = 1别的:每个客户 = 0客户数 = 0对于每个 popn.people:如果每个客户 == 1:客户数量 += 1返回客户数t0 = timer_func()对于 xrange(20000) 中的 i:x = f_wo_append(popn)t1 = timer_func()打印 t1-t0

编辑 4:查看 John Machin 和 TryPyPy 的答案

由于这里的编辑和更新太多了，第一次来到这里的人可能会有点困惑.请参阅 John Machin 和 TryPyPy 的答案.这两者都可以帮助大大提高代码的速度.我很感谢他们和其他提醒我 append 缓慢的人.由于在这种情况下我将使用 John Machin 的解决方案而不是使用 numpy 来生成实用程序，因此我接受他的回答作为答案.但是，我也非常感谢 TryPyPy 指出的方向.

解决方案

请考虑缩减您的 f_wo_append 功能:

def f_wo_append():'''无附加功能'''P = 75客户数 = 0对于 popn.people 中的人:person.customer = iscust = person.utility >= P客户数量 += iscust返回客户数

编辑以回应 OP 的评论"这让情况变得更糟！修剪后的版本比我上面发布的版本花费的时间多 4 倍.""

没有办法可以多花 4 倍"(5 倍?)……这是我的代码，正如我所建议的，它展示了无附加"情况的显着减少，并且还引入了在with append"情况下的显着改进.

import random # 而不是 numpy导入时间timer_func = time.clock # 在 Windows 上更好，在 *x 平台上使用 time.time类人(对象):def __init__(self, util):self.utility = utilself.customer = 0类人口(对象):def __init__(self, numpeople):随机种子(1)self.people = [Person(random.uniform(0, 300)) for i in xrange(numpeople)]self.cus = []self.noncus = []def f_w_append(popn):'''带附加功能'''P = 75cus = []noncus = []对于每个 popn.people:如果 per.utility >= P:每个客户 = 1cus.append(per)别的:每个客户 = 0noncus.append(per)popn.cus = cus # 从 OP 的代码中省略popn.noncus = noncus # 从 OP 的代码中省略返回 len(cus)def f_w_append2(popn):'''带附加功能'''P = 75popn.cus = []popn.noncus = []cusapp = popn.cus.appendnoncusapp = popn.noncus.append对于每个 popn.people:如果 per.utility >= P:每个客户 = 1cusapp(每个)别的:每个客户 = 0noncusapp(每)返回 len(popn.cus)def f_wo_append(popn):'''无附加功能'''P = 75对于每个 popn.people:如果 per.utility >= P:每个客户 = 1别的:每个客户 = 0客户数 = 0对于每个 popn.people:如果每个客户 == 1:客户数量 += 1返回客户数def f_wo_append2(popn):'''无附加功能'''P = 75客户数 = 0对于 popn.people 中的人:person.customer = iscust = person.utility >= P客户数量 += iscust返回客户数如果 __name__ == "__main__":导入系统popsize, which, niter = map(int, sys.argv[1:4])pop = 人口(popsize)func = (f_w_append, f_w_append2, f_wo_append, f_wo_append2)[其中]t0 = timer_func()对于 _unused 在 xrange(niter) 中:nc = 函数(弹出)t1 = timer_func()打印 "popsize=%d func=%s niter=%d nc=%d seconds=%.2f" % (popsize, func.__name__, niter, nc, t1 - t0)

这是运行它的结果(Python 2.7.1、Windows 7 Pro、Intel Core i3 CPU 540 @ 3.07 GHz"):

C:\junk>\python27\python ncust.py 300 0 80000popsize=300 func=f_w_append niter=80000 nc=218 秒=5.48C:\垃圾>\python27\python ncust.py 300 1 80000popsize=300 func=f_w_append2 niter=80000 nc=218 秒=4.62C:\垃圾>\python27\python ncust.py 300 2 80000popsize=300 func=f_wo_append niter=80000 nc=218 秒=5.55C:\垃圾>\python27\python ncust.py 300 3 80000popsize=300 func=f_wo_append2 niter=80000 nc=218 秒=4.29

编辑 3 为什么 numpy 需要更长的时间:

<预><代码>>>>导入 numpy>>>utils = numpy.random.uniform(0, 300, 10)>>>打印 repr(utils[0])42.777972538362874>>>类型(实用程序 [0])<输入'numpy.float64'>

这就是为什么我的 f_wo_append2 函数花费了 4 倍的时间:

<预><代码>>>>x = 实用程序 [0]>>>类型(x)<输入'numpy.float64'>>>>类型(x >= 75)<输入'numpy.bool_'># iscust 指的是一个 numpy.bool_>>>类型(0 + (x >= 75))<输入'numpy.int32'># numcustomers 最终指向一个 numpy.int32>>>

经验证据表明，这些自定义类型在用作标量时并没有那么快……也许是因为它们每次使用时都需要重置浮点硬件.适用于大数组，不适用于标量.

您是否使用任何其他 numpy 功能?如果没有，只需使用 random 模块.如果您对 numpy 有其他用途，您可能希望在填充设置期间将 numpy.float64 强制为 float.

I have some python code that has many classes. I used cProfile to find that the total time to run the program is 68 seconds. I found that the following function in a class called Buyers takes about 60 seconds of those 68 seconds. I have to run the program about 100 times, so any increase in speed will help. Can you suggest ways to increase the speed by modifying the code? If you need more information that will help, please let me know.

def qtyDemanded(self, timePd, priceVector):
    '''Returns quantity demanded in period timePd. In addition,
    also updates the list of customers and non-customers.

    Inputs: timePd and priceVector
    Output: count of people for whom priceVector[-1] < utility
    '''

    ## Initialize count of customers to zero
    ## Set self.customers and self.nonCustomers to empty lists
    price = priceVector[-1]
    count = 0
    self.customers = []
    self.nonCustomers = []


    for person in self.people:
        if person.utility >= price:             
            person.customer = 1
            self.customers.append(person)
        else:
            person.customer = 0
            self.nonCustomers.append(person)

    return len(self.customers)

self.people is a list of person objects. Each person has customer and utility as its attributes.

EDIT - responsed added

-------------------------------------

Thanks so much for the suggestions. Here is the response to some questions and suggestions people have kindly made. I have not tried them all, but will try others and write back later.

(1) @amber - the function is accessed 80,000 times.

(2) @gnibbler and others - self.people is a list of Person objects in memory. Not connected to a database.

(3) @Hugh Bothwell

cumtime taken by the original function - 60.8 s (accessed 80000 times)

cumtime taken by the new function with local function aliases as suggested - 56.4 s (accessed 80000 times)

(4) @rotoglup and @Martin Thomas

I have not tried your solutions yet. I need to check the rest of the code to see the places where I use self.customers before I can make the change of not appending the customers to self.customers list. But I will try this and write back.

(5) @TryPyPy - thanks for your kind offer to check the code.

Let me first read a little on the suggestions you have made to see if those will be feasible to use.

EDIT 2 Some suggested that since I am flagging the customers and noncustomers in the self.people, I should try without creating separate lists of self.customers and self.noncustomers using append. Instead, I should loop over the self.people to find the number of customers. I tried the following code and timed both functions below f_w_append and f_wo_append. I did find that the latter takes less time, but it is still 96% of the time taken by the former. That is, it is a very small increase in the speed.

@TryPyPy - The following piece of code is complete enough to check the bottleneck function, in case your offer is still there to check it with other compilers.

Thanks again to everyone who replied.

import numpy

class person(object):
    def __init__(self, util):
        self.utility = util
        self.customer = 0

class population(object):
    def __init__(self, numpeople):
        self.people = []
        self.cus = []
        self.noncus = []
        numpy.random.seed(1)
        utils = numpy.random.uniform(0, 300, numpeople)
        for u in utils:
            per = person(u)
            self.people.append(per)

popn = population(300)

def f_w_append():
    '''Function with append'''
    P = 75
    cus = []
    noncus = []
    for per in popn.people:
        if  per.utility >= P:
            per.customer = 1
            cus.append(per)
        else:
            per.customer = 0
            noncus.append(per)
    return len(cus)

def f_wo_append():
    '''Function without append'''
    P = 75
    for per in popn.people:
        if  per.utility >= P:
            per.customer = 1
        else:
            per.customer = 0

    numcustomers = 0
    for per in popn.people:
        if per.customer == 1:
            numcustomers += 1                
    return numcustomers

EDIT 3: It seems numpy is the problem

This is in response to what John Machin said below. Below you see two ways of defining Population class. I ran the program below twice, once with each way of creating Population class. One uses numpy and one does not use numpy. The one without numpy takes similar time as John found in his runs. One with numpy takes much longer. What is not clear to me is that the popn instance is created before time recording begins (at least that is what it appears from the code). Then, why is numpy version taking longer. And, I thought numpy was supposed to be more efficient. Anyhow, the problem seems to be with numpy and not so much with the append, even though it does slow down things a little. Can someone please confirm with the code below? Thanks.

import random # instead of numpy
import numpy
import time
timer_func = time.time # using Mac OS X 10.5.8

class Person(object):
    def __init__(self, util):
        self.utility = util
        self.customer = 0

class Population(object):
    def __init__(self, numpeople):
        random.seed(1)
        self.people = [Person(random.uniform(0, 300)) for i in xrange(numpeople)]
        self.cus = []
        self.noncus = []   

# Numpy based    
# class Population(object):
#     def __init__(self, numpeople):
#         numpy.random.seed(1)
#         utils = numpy.random.uniform(0, 300, numpeople)
#         self.people = [Person(u) for u in utils]
#         self.cus = []
#         self.noncus = []    


def f_wo_append(popn):
    '''Function without append'''
    P = 75
    for per in popn.people:
        if  per.utility >= P:
            per.customer = 1
        else:
            per.customer = 0

    numcustomers = 0
    for per in popn.people:
        if per.customer == 1:
            numcustomers += 1                
    return numcustomers



t0 = timer_func()
for i in xrange(20000):
    x = f_wo_append(popn)
t1 = timer_func()
print t1-t0

Edit 4: See the answers by John Machin and TryPyPy

Since there have been so many edits and updates here, those who find themselves here for the first time may be a little confused. See the answers by John Machin and TryPyPy. Both of these can help in improving the speed of the code substantially. I am grateful to them and others who alerted me to slowness of append. Since, in this instance I am going to use John Machin's solution and not use numpy for generating utilities, I am accepting his response as an answer. However, I really appreciate the directions pointed out by TryPyPy also.

解决方案

Please consider trimming down your f_wo_append function:

def f_wo_append():
    '''Function without append'''
    P = 75
    numcustomers = 0
    for person in popn.people:
        person.customer = iscust = person.utility >= P
        numcustomers += iscust
    return numcustomers

Edit in response to OP's comment """This made it a lot worse! The trimmed version takes 4 times more time than the version I have posted above. """

There is no way that that could take "4 times more" (5 times?) ... here is my code, which demonstrates a significant reduction in the "without append" case, as I suggested, and also introduces a significant improvement in the "with append" case.

import random # instead of numpy
import time
timer_func = time.clock # better on Windows, use time.time on *x platform

class Person(object):
    def __init__(self, util):
        self.utility = util
        self.customer = 0

class Population(object):
    def __init__(self, numpeople):
        random.seed(1)
        self.people = [Person(random.uniform(0, 300)) for i in xrange(numpeople)]
        self.cus = []
        self.noncus = []        

def f_w_append(popn):
    '''Function with append'''
    P = 75
    cus = []
    noncus = []
    for per in popn.people:
        if  per.utility >= P:
            per.customer = 1
            cus.append(per)
        else:
            per.customer = 0
            noncus.append(per)
    popn.cus = cus # omitted from OP's code
    popn.noncus = noncus # omitted from OP's code
    return len(cus)

def f_w_append2(popn):
    '''Function with append'''
    P = 75
    popn.cus = []
    popn.noncus = []
    cusapp = popn.cus.append
    noncusapp = popn.noncus.append
    for per in popn.people:
        if  per.utility >= P:
            per.customer = 1
            cusapp(per)
        else:
            per.customer = 0
            noncusapp(per)
    return len(popn.cus)    

def f_wo_append(popn):
    '''Function without append'''
    P = 75
    for per in popn.people:
        if  per.utility >= P:
            per.customer = 1
        else:
            per.customer = 0

    numcustomers = 0
    for per in popn.people:
        if per.customer == 1:
            numcustomers += 1                
    return numcustomers

def f_wo_append2(popn):
    '''Function without append'''
    P = 75
    numcustomers = 0
    for person in popn.people:
        person.customer = iscust = person.utility >= P
        numcustomers += iscust
    return numcustomers    

if __name__ == "__main__":
    import sys
    popsize, which, niter = map(int, sys.argv[1:4])
    pop = Population(popsize)
    func = (f_w_append, f_w_append2, f_wo_append, f_wo_append2)[which]
    t0 = timer_func()
    for _unused in xrange(niter):
        nc = func(pop)
    t1 = timer_func()
    print "popsize=%d func=%s niter=%d nc=%d seconds=%.2f" % (
        popsize, func.__name__, niter, nc, t1 - t0)

and here are the results of running it (Python 2.7.1, Windows 7 Pro, "Intel Core i3 CPU 540 @ 3.07 GHz"):

C:\junk>\python27\python ncust.py 300 0 80000
popsize=300 func=f_w_append niter=80000 nc=218 seconds=5.48

C:\junk>\python27\python ncust.py 300 1 80000
popsize=300 func=f_w_append2 niter=80000 nc=218 seconds=4.62

C:\junk>\python27\python ncust.py 300 2 80000
popsize=300 func=f_wo_append niter=80000 nc=218 seconds=5.55

C:\junk>\python27\python ncust.py 300 3 80000
popsize=300 func=f_wo_append2 niter=80000 nc=218 seconds=4.29

Edit 3 Why numpy takes longer:

>>> import numpy
>>> utils = numpy.random.uniform(0, 300, 10)
>>> print repr(utils[0])
42.777972538362874
>>> type(utils[0])
<type 'numpy.float64'>

and here's why my f_wo_append2 function took 4 times longer:

>>> x = utils[0]
>>> type(x)
<type 'numpy.float64'>
>>> type(x >= 75) 
<type 'numpy.bool_'> # iscust refers to a numpy.bool_
>>> type(0 + (x >= 75)) 
<type 'numpy.int32'> # numcustomers ends up referring to a numpy.int32
>>>

The empirical evidence is that these custom types aren't so fast when used as scalars ... perhaps because they need to reset the floating-point hardware each time they are used. OK for big arrays, not for scalars.

Are you using any other numpy functionality? If not, just use the random module. If you have other uses for numpy, you may wish to coerce the numpy.float64 to float during the population setup.

这篇关于提高python代码的速度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！