如何保留大量实体(JPA) [英] How to persist a lot of entities (JPA)

问题描述

我需要处理CSV文件，并且对于每个记录(行)都保留一个实体.现在，我这样做:

I need to process a CSV file and for each record (line) persist an entity. Right now, I do it this way:

while ((line = reader.readNext()) != null) {
    Entity entity = createEntityObject(line);
    entityManager.save(entity);
    i++;
}

其中，save(Entity)方法基本上只是一个EntityManager.merge()调用. CSV文件中大约有20,000个实体(行).这是一种有效的方法吗?它似乎很慢.使用EntityManager.persist()会更好吗?这个解决方案有任何缺陷吗?

where the save(Entity) method is basically just an EntityManager.merge() call. There are about 20,000 entities (lines) in the CSV file. Is this an effective way to do it? It seems to be quite slow. Would it be better to use EntityManager.persist()? Is this solution flawed in any way?

编辑

这是一个漫长的过程(超过400秒钟)，我使用persist和merge尝试了两种解决方案.两者花费的时间大致相同(459s与443s).问题是，像这样一个接一个地保存实体是否最佳.据我所知，Hibernate(这是我的JPA提供程序)确实实现了某些缓存/刷新功能，因此我不必为此担心.

EDIT

It's a lengthy process (over 400s) and I tried both solutions, with persist and merge. Both take approximately the same amount of time to complete (459s vs 443s). The question is if saving the entities one by one like this is optimal. As far as I know, Hibernate (which is my JPA provider) does implement some cache/flush functionality so I shouldn't have to worry about this.

推荐答案

JPA API并未为您提供最佳选择.根据您要执行此操作的速度，您将不得不寻找特定于ORM的选项-在您的情况下为休眠状态.

The JPA API doesn't provide you all the options to make this optimal. Depending on how fast you want to do this you are going to have to look for ORM specific options - Hibernate in your case.

要检查的事情:

1. 检查您是否正在使用一次交易(是的，显然您对此有把握)
2. 检查您的JPA提供程序(休眠)是否正在使用JDBC批处理API(请参阅:hibernate.jdbc.batch_size)
3. 检查是否可以绕过获取生成的密钥(取决于db/jdbc驱动程序，您从中获得多少好处-请参阅:hibernate.jdbc.use_getGeneratedKeys)
4. 检查是否可以绕过级联逻辑(这样做只能带来最小的性能优势)

因此在Ebean ORM中，这将是:

So in Ebean ORM this would be:

    EbeanServer server = Ebean.getServer(null);

    Transaction transaction = server.beginTransaction();
    try {
        // Use JDBC batch API with a batch size of 100
        transaction.setBatchSize(100);
        // Don't bother getting generated keys
        transaction.setBatchGetGeneratedKeys(false);
        // Skip cascading persist 
        transaction.setPersistCascade(false);

        // persist your beans ...
        Iterator<YourEntity> it = null; // obviously should not be null 
        while (it.hasNext()) {
            YourEntity yourEntity = it.next();
            server.save(yourEntity);
        }

        transaction.commit();
    } finally {
        transaction.end();
    }

哦，如果您通过原始JDBC执行此操作，则会跳过ORM开销(减少了对象创建/垃圾收集等操作)，因此我不会忽略该选项.

Oh, and if you do this via raw JDBC you skip the ORM overhead (less object creation / garbage collection etc) - so I wouldn't ignore that option.

是的，这不能回答您的问题，但可能有助于您搜索更多ORM特定的批处理插入调整.

So yes, this doesn't answer your question but might help your search for more ORM specific batch insert tweaks.

这篇关于如何保留大量实体(JPA)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！