Windows中的Python:使用pyodbc进行大量插入会导致内存泄漏 [英] Python in Windows: large number of inserts using pyodbc causes memory leak

问题描述

我正在尝试在Windows上使用python填充MS SQL 2005数据库.我要插入数百万行，到700万行，我将使用近1 GB的内存.每插入10万行，下面的测试就会占用4兆的RAM:

I am trying to populate a MS SQL 2005 database using python on windows. I am inserting millions of rows, and by 7 million I am using almost a gigabyte of memory. The test below eats up 4 megs of RAM for each 100k rows inserted:

import pyodbc
connection=pyodbc.connect('DRIVER={SQL Server};SERVER=x;DATABASE=x;UID=x;PWD=x')
cursor=connection.cursor()
connection.autocommit=True
while 1:
    cursor.execute("insert into x (a,b,c,d, e,f) VALUES (?,?,?,?,?,?)",1,2,3,4,5,6)
mdbconn.close()

hack解决方案:我最终使用多处理模块生成了一个新进程以返回内存.仍然对为什么以这种方式插入行会占用大量内存感到困惑.有什么想法吗?

Hack solution: I ended up spawning a new process using the multiprocessing module to return memory. Still confused about why inserting rows in this way consumes so much memory. Any ideas?

推荐答案

我遇到了同样的问题，它看起来像是带有参数化插入内容的pyodbc问题:http://code.google.com/p/pyodbc/issues/detail?id=145

I had the same issue, and it looks like a pyodbc issue with parameterized inserts: http://code.google.com/p/pyodbc/issues/detail?id=145

临时填充到填充有VALUES子句的静态插入可以消除泄漏，直到我尝试从当前源进行构建为止.

Temporarily switching to a static insert with the VALUES clause populated eliminates the leak, until I try a build from the current source.

这篇关于Windows中的Python:使用pyodbc进行大量插入会导致内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！