使用SQLalchemy读取大文件 [英] Reading large files using SQLalchemy

问题描述

我试图使用SQLalchemy读取一个200 MB的csv文件。每行有大约30列，其中，我只使用下面的代码8列。但是，代码运行真的很慢！有办法改善这个吗？我想使用地图/列表推导或其他技术。正如你所说，我是一个新手。谢谢你的帮助。

I am trying to read a 200 MB csv file using SQLalchemy. Each line has about 30 columns, of which, I use only 8 columns using the code below. However, the code runs really slow! Is there a way to improve this? I would like to use map/list comprehension or other techniques. As you an tell, I am a newbie. Thanks for your help.

for ddata in dread:        
    record = DailyData()
    record.set_campaign_params(pdata) #Pdata is assigned in the previous step         
    record.set_daily_data(ddata) #data is sent to a class method where only 8 of 30 items in the list are used       
    session.add(record)
    session.commit() #writing to the SQL database.  

推荐答案

提交或只是刷新每1000左右：

don't commit on every record. commit or just flush every 1000 or so:

for i, data in enumerate(csv_stuff):
    rec = MyORMObject()
    rec.set_stuff(data)
    session.add(rec)
    if i % 1000 == 0:
        session.flush()
session.commit() # flushes everything remaining + commits

如果仍然给你问题，我的帖子在如何配置一个SQLAlchemy驱动的应用程序？

if that's still giving you problems then do some basic profiling, see my post at How can I profile a SQLAlchemy powered application?

这篇关于使用SQLalchemy读取大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！