如何填充Django sqlite3数据库 [英] How to populate a Django sqlite3 database

问题描述

我的计划是分批从网站上收集律师传记数据，并将每批文件转换为.csv文件，然后转换为json，然后再将其加载到Django数据库中.

My plan is to collect lawyer biography data from websites in batches and convert each batch into a .csv file, then to json, and then load each into a Django database.

请让我知道如何以最佳方式完成这项任务.

Please let me know how to achieve this task the best way.

推荐答案

只需直接加载数据库.批量从网站收集数据，直接加载SQlite3.只需编写使用Django ORM的简单批处理应用程序即可.从网站收集数据并立即加载SQLite3.不要创建CSV.不要创建JSON.不要创建中间结果.不要做任何额外的工作.

Just load the database directly. Collect data from websites in batches, loading the SQlite3 directly. Just write simple batch applications that use the Django ORM. Collect data from websites and load SQLite3 immediately. Do not create CSV. Do not create JSON. Do not create intermediate results. Do not do any extra work.

编辑.

from myapp.models import MyModel
import urllib2

with open("sourceListOfURLs.txt", "r" ) as source:
    for aLine in source:
        for this, the, the_other in someGenerator( aLine ):
            object= MyModel.objects.create( field1=this, field2=that, field3=the_other )
            object.save()

def someGenerator( url ):
    # open the URL with urllib2
    # parse the data with BeautifulSoup
    yield this, that, the_other 

这篇关于如何填充Django sqlite3数据库的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！