安卓的java：内存不足而插入一个巨大的名单到数据库SQL [英] Android java: OutOfMemory while inserting a huge List into the database SQL

问题描述

我只是工作的一个Android项目，其中一个ORM greendao使用。它让我插入了一些实体（这显然是对象）到数据库一次（在一个事务中）。在实践中有一个方法

I'm just working on an Android Project where an ORM greendao is used. It allows me to insert a number of entities (which are obviously objects) into the database at once (in one transaction). In practice there's a method

insertOrReplaceInTx（...）

insertOrReplaceInTx(...)

这需要作为一个参数的集合，给定对象的列表。问题是，我的名单上有某种12000对象和时间插入导致了时间的内存不足异常。我想了一个巧妙的方法来解决问题，prevent未来OOM。这在我脑海中唯一的想法就是分裂很大成排序子集合（假设每个500元），并承诺在一个循环做几个小犯，而不是一个巨大的。关于它的坏的事情是，它需要更长的时间来这里插入记录和时间的问题。最后，我不知道，如果让犯后提交不杀堆反正。也许调用sleep（）方法之间，让GC清除堆......但在我看来是一种丑陋的解决方案。

which takes as a parameter a collection, given a List of objects. The problem is that my list has sort of 12000 objects and an insert leads from time to time to an OutOfMemory Exception. I'm thinking of a smart way to solve the problem and prevent future OOM. The only idea which comes to my mind is to split the huge collection into sort of subcollections (let's say 500 elements each) and commit in a loop making several small commits instead of one huge. The bad thing about it is that it takes longer to insert the records and time matters here. In the end I'm not sure if making commit after commit doesn't kill the heap anyway. Maybe call a sleep() method in between to let GC clear the heap ...but it looks to me as a kind of ugly solution.

任何智能的想法在你的身边？ 在此先感谢

any smart ideas on your side? thanks in advance

推荐答案

我正在使用greendao 1.3.1项目。一些含有20多万的实体表（不包含很多的属性）。

I am working on a project using greendao 1.3.1. Some of the tables containing about 200000 entities (not containing a lot of properties).

我读的实体CSV和加快速度我开发了一个小巧的解决方案，这也可以帮助你的OOM问题的问题。

I read the entities from csv and to speed things up I developed a small solution, which might also help with your OOM-issue.

有关说明：

greendao使用一个缓存后的每个插入更新实体，以获取该行-ID，可能插入实体到它的高速缓存。在那个greendao的顶部开始交易，如果你调用一个插入或更新的方法，如果有尚未事务。这大大降低了大头-inserts，增加了内存使用量，也降低了速度。

greendao uses a cache and after each insert it updates the entity to get the row-id and probably inserts the entity into its cache. On top of that greendao starts a transaction if you call an insert or an update method and if there isn't already a transaction. This slows down "bulk"-inserts and increases the memory usage and also reduces speed.

我所做的：

要拧紧的事情了，我开始交易之前，我没有任何插件。这样greendao不会启动一个事务上的每个插入和所有的插入和更新，其中有关于数据一致性的额外利益相同的事务。 你可以用code是这样的：

To fasten things up I started a transaction before I did any insert. This way greendao will not start a transaction on every insert and all inserts and updates are in the same transaction which has additional benefits concerning data consistency. You can use code like this:

SQLiteDatabase db = dao.getDatabase();
db.beginTransaction();

try {
    // do all your inserts and so on here.
    db.setTransactionSuccessful();
} catch (Exception ex) {
} finally {
    db.endTransaction();
}

不过，这不会帮助你的OOM-的问题呢。

But this won't help you with your OOM-problem yet.

解决方法1

如果你不想惹greendao- code，你可以在一段时间发出 DaoSession.clear（）每一次。 这绝对是简单的解决方案，但会比方案二少高性能的。

If you don't want to mess with the greendao-code you can issue a DaoSession.clear() every once in a while. This is definitely the simple solution, but will be less performant than solution 2.

解决方案2

要由updateing prevent greendao并插入实体到它的缓存，你可以用这个$ C替换方法专用长executeInsert（T实体，SQLiteStatement语句） $ C中的 AbstractDao.java

To prevent greendao from updateing and inserting the entity into its cache you can replace the method private long executeInsert(T entity, SQLiteStatement stmt) with this code in AbstractDao.java:

/**
 * Insert an entity into the table associated with a concrete DAO.
 * 
 * @return row ID of newly inserted entity
 */
public long insertOrReplace(T entity, boolean update) {
    return executeInsert(entity, statements.getInsertOrReplaceStatement(), update);
}

private long executeInsert(T entity, SQLiteStatement stmt) {
    return executeInsert(entity, stmt, true);
}

private long executeInsert(T entity, SQLiteStatement stmt, boolean update) {
    long rowId;
    if (db.isDbLockedByCurrentThread()) {
        synchronized (stmt) {
            bindValues(stmt, entity);
            rowId = stmt.executeInsert();
        }
    } else {
        // Do TX to acquire a connection before locking the stmt to avoid deadlocks
        db.beginTransaction();
        try {
            synchronized (stmt) {
                bindValues(stmt, entity);
                rowId = stmt.executeInsert();
            }
            db.setTransactionSuccessful();
        } finally {
            db.endTransaction();
        }
    }
    if (update) {
        updateKeyAfterInsertAndAttach(entity, rowId, true);
    }
    return rowId;
}

这篇关于安卓的java：内存不足而插入一个巨大的名单到数据库SQL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！