Python在遍历大列表时很慢 [英] Python is slow when iterating over a large list

问题描述

我目前正在使用pyodbc从数据库中选择一个大的列表。然后将结果复制到一个大列表，然后我试图迭代列表。在我放弃python之前，尝试在C＃中创建它，我想知道是否有一些我做错了。

  clientItems.execute（Select id from largetable where year =？，year）; 
 allIDRows = clientItemsCursor.fetchall（）#takes也许8秒。 
 
对于allIDRows中的clientItemrow：
 aID = str（clientItemRow [0]）
＃使用str做一些事情 - 删除，因为我试图确定什么是慢
 count = count + 1 
  

更多信息：

$ b b

• for循环目前以每秒5个循环的速度运行，这对我来说似乎很慢。


• 所选的总行数为〜489,000。


• 其运行的计算机具有大量RAM和CPU。它似乎只运行一个或两个内核，而ram是4GB的1.72GB。

任何人都可以告诉我什么错了？

感谢

解决方案

不是与Python本地列表慢 - 但也许ODBC的驱动程序返回一个懒惰的对象，试图是聪明的，但只是慢。尝试只做

allIDRows = list（clientItemsCursor.fetchall（））

$

（如果您开始在中间插入内容，Python列表可能会变慢，而只是迭代过大的列表应该快）

I am currently selecting a large list of rows from a database using pyodbc. The result is then copied to a large list, and then i am trying to iterate over the list. Before I abandon python, and try to create this in C#, I wanted to know if there was something I was doing wrong.

clientItems.execute("Select ids from largetable where year =?", year);
allIDRows = clientItemsCursor.fetchall() #takes maybe 8 seconds.

for clientItemrow in allIDRows:
    aID = str(clientItemRow[0])
    # Do something with str -- Removed because I was trying to determine what was slow
    count = count+1

Some more information:

• The for loop is currently running at about 5 loops per second, and that seems insanely slow to me.
• The total rows selected is ~489,000.
• The machine its running on has lots of RAM and CPU. It seems to only run one or two cores, and ram is 1.72GB of 4gb.

Can anyone tell me whats wrong? Do scripts just run this slow?

Thanks

解决方案

This should not be slow with Python native lists - but maybe ODBC's driver is returning a "lazy" object that tries to be smart but just gets slow. Try just doing

allIDRows = list(clientItemsCursor.fetchall())

in your code and post further benchmarks.

(Python lists can get slow if you start inserting things in its middle, but just iterating over a large list should be fast)

这篇关于Python在遍历大列表时很慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！