Grails，使用withTransaction插入大量数据会导致OutOfMemoryError [英] Grails, Inserting lots of data using withTransaction results in OutOfMemoryError

问题描述

我正在使用Grails 1.1 beta2。我需要将大量数据导入到我的Grails应用程序中。如果我反复实例化一个grails域的类然后保存它，那么性能会慢得令人无法接受。例如，从电话簿中导入人员：

  for（传统电话簿中的每个人）{
 //从传统电话簿人员构建新的Grails域类
 Person person = new Person（...）
 person.save（）
} 
  

事实证明，这会非常缓慢。 Grails邮件列表中的某个人建议在交易中加入保存。所以现在我有：

$ p $ 列表batch = new ArrayList（）
（在传统电话簿中的每个人）{
//从传统电话簿人员构建新的Grails域类
Person person = new Person（...）
batch.add（person）
if（batch.size（ ）> 500）{
Person.withTransaction {
for（Person p：batch）
p.save（）
batch.clear（）
}

$ b //保存任何剩余的
（Person p：batch）
p.save（）


这起作用必须更快，至少在最初时。每笔交易可保存500条记录。随着时间的推移，交易时间越来越长。前几笔交易大约需要5秒钟，然后从那里开始。经过大约100次交易后，每一次交易都花费了一分钟，这再次令人无法接受。更糟的是，最终Grails最终会耗尽Java堆内存。我可以增加JVM堆大小，但这只会延迟 OutOfMemoryError 异常。

任何想法为什么会这样？这就像有一些内部资源没有被释放。性能越来越差，内存正在被保留，然后系统最终耗尽内存。 b
$ b

根据 Grails文档， withTransaction 将闭包传递给Spring的 TransactionStatus 对象。我无法在 TransactionStatus 中找到任何关闭/结束交易的内容。

编辑： grails console ）运行了这个程序。

编辑：

 抛出的异常：Java堆空间
 
 java.lang .OutOfMemoryError：Java堆空间
位于org.hibernate.util.IdentityMap.entryArray（IdentityMap.java:194）
位于org.hibernate.util.IdentityMap.concurrentEntries（IdentityMap.java:59）
 at org.hibernate.event.def.AbstractFlushingEventListener.prepareEntityFlushes（AbstractFlushingEventListener.java:113）
 at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions（AbstractFlushingEventListener.java:65）
 at org .hibernate.event.def.DefaultFlushEventListener.onFlush（DefaultFlushEventListener.java:26）
 at org.hibernate.impl.SessionImpl.flush（SessionImpl.java:1000）
 org.hibernate.impl.SessionImpl.managedFlush（SessionImpl.java:338）
 at org.hibernate.transaction.JDBCTransaction.commit（JDBCTransaction.java:106）
 at org.springframework.orm.hibernate3 .HibernateTransactionManager.doCommit（HibernateTransactionManager.java:655）
 at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit（AbstractPlatformTransactionManager.java:732）
 at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit （AbstractPlatformTransactionManager.java:701）
 at org.springframework.transaction.support.TransactionTemplate.execute（TransactionTemplate.java:140）
  

解决方案

这是所有hibernate应用程序的常见问题，它是由hibernate会话的增长引起的。我在猜测，grails控制台持有一个hibernate会话，它以类似于我知道它用于正常web请求的'打开会话'模式的方式为您打开。

解决方案是获取当前会话并在每批后清除它。我不确定你如何使用控制台获取spring bean，通常用于控制器或服务，您只需将它们声明为成员。然后，您可以使用 sessionFactory.getCurrentSession（）获取当前会话。为了清除它，只需调用 session.clear（），或者如果您选择使用 session.evict（Object）为每个 Person 对象。

用于控制器/服务：

  class FooController {
 def sessionFactory 
 
 def doStuff = {
列表batch = new ArrayList（）
（传统电话簿中的每个人）{
 //构建新的Grails （...）
 batch.add（person）
 if（batch.size（）> 500）{
 Person.withTransaction {
 for（Person p：batch）
 p.save（）
 batch.clear（）
} 
 //在此清除会话。 
 sessionFactory.getCurrentSession（）。clear（）; 
 
 
 //保存任何剩余的
（Person p：batch）
 p.save（）
} 
} 
} 
  

希望这有助于您。

I'm using Grails 1.1 beta2. I need to import a large amount of data into my Grails application. If I repeatedly instantiate a grails domain class and then save it, the performance is unacceptably slow. Take for example importing people from a phone book:

for (each person in legacy phone book) {
    // Construct new Grails domain class from legacy phone book person
    Person person = new Person(...)
    person.save()
}

This turns out be painfully slow. Someone on the Grails mailing list suggest batching up saves in a transaction. So now I have:

List batch = new ArrayList()
for (each person in legacy phone book) {
    // Construct new Grails domain class from legacy phone book person
    Person person = new Person(...)
    batch.add(person)
    if (batch.size() > 500) {
        Person.withTransaction {
            for (Person p: batch)
                p.save()
            batch.clear()
        }
    }
}
// Save any remaining
for (Person p: batch)
    p.save()

This works must faster, at least initially. Each transaction saves 500 records. As time goes on, the transactions take longer and longer. The first few transactions takes about 5 seconds, then it just creeps from there. After about 100 transactions, each one takes over a minute, which is once again unacceptable. Worse is that eventually Grails will eventually run out of Java heap memory. I can increase the JVM heap size, but that just delays the OutOfMemoryError exception.

Any ideas why this is? It's like there some internal resource not being released. The performance gets worse, memory is being held on to, and then eventually the system runs out of memory.

According to the Grails documentation, withTransaction passes the closure to Spring's TransactionStatus object. I couldn't find anything in TransactionStatus to close/end the transaction.

Edit: I'm running this from Grails' Console (grails console)

Edit: Here's the out of memory exception:

Exception thrown: Java heap space

java.lang.OutOfMemoryError: Java heap space
    at org.hibernate.util.IdentityMap.entryArray(IdentityMap.java:194)
    at org.hibernate.util.IdentityMap.concurrentEntries(IdentityMap.java:59)
    at org.hibernate.event.def.AbstractFlushingEventListener.prepareEntityFlushes(AbstractFlushingEventListener.java:113)
    at org.hibernate.event.def.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:65)
    at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:26)
    at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
    at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
    at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
    at org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:655)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:732)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)

解决方案

This is a common issue with all hibernate applications and it is caused by the growth of the hibernate session. I'm guessing that the grails console holds a hibernate session open for you in a similar way to the 'open session in view' pattern that I know it uses in for normal web requests.

The solution is to get hold of the current session and clear it after each batch. I'm not sure how you get hold of spring bean using the console, normally for controllers or services you just declare them as members. Then you can get the current session with sessionFactory.getCurrentSession(). In order to clear it just call session.clear(), or if you what to be selective use session.evict(Object) for each Person object.

for a controller/service:

class FooController {
    def sessionFactory

    def doStuff = {
        List batch = new ArrayList()
        for (each person in legacy phone book) {
            // Construct new Grails domain class from legacy phone book person
            Person person = new Person(...)
            batch.add(person)
            if (batch.size() > 500) {
                Person.withTransaction {
                    for (Person p: batch)
                        p.save()
                    batch.clear()
                }
                // clear session here.
                sessionFactory.getCurrentSession().clear();
            }
        }
        // Save any remaining
        for (Person p: batch)
            p.save()
        }
    }
}

Hope this helps.

这篇关于Grails，使用withTransaction插入大量数据会导致OutOfMemoryError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！