在处理大量元素时，休眠内存异常 [英] Hibernate out of memory exception while processing large collection of elements

问题描述

我正在尝试处理重磅元素（图像）的集合。收集的大小在8000到50000个条目之间变化。但由于某些原因，处理1800-1900条目后，我的程序将落入java.lang.OutOfMemoryError：Java堆空间。

在我每次调用session.getTransaction （）.commit（）程序应该释放堆内存，但看起来从未发生过。
我做错了什么？这里是代码：

pre $ private static void loadImages（LoadStrategy loadStrategy）throws IOException {
log.info（图片为：+ loadStrategy.getPageType（））;

会话会话= sessionFactory.openSession（）;
session.setFlushMode（FlushMode.COMMIT）;
Query query = session.createQuery（from PageRaw where pageType =：pageType and pageStatus =：pageStatus and sessionId = 1）;
query.setString（pageStatus，PageStatus.SUCCESS.name（））;
query.setString（pageType，loadStrategy.getPageType（）。name（））;
query.setMaxResults（50）;

列表< PageRaw>页;
int resultNum = 0;

do {

session.getTransaction（）。begin（）;

log.info（Get pages statring form+ resultNum +position）;
query.setFirstResult（resultNum）;
resultNum + = 50;
pages = query.list（）;
log.info（Found+ pages.size（）+pages）;


（PageRaw pr：pages）{
Set< String> imageUrls = new HashSet<>（）;
for（UrlLocator imageUrlLocator：loadStrategy.getImageUrlLocators（））{
imageUrls.addAll（
imageUrlLocator.locateUrls（StringConvector.toString（pr.getSourceHtml（）））
）;
}

removeDeletedImageRaws（pr.getImages（），imageUrls）;
loadNewImageRaws（pr.getImages（），imageUrls）;
}

session.getTransaction（）。commit（）;

} while（pages.size（）> 0）;

session.close（）;
}


解决方案

使用清除清空：

• 执行所有针对数据库的挂起语句（它将内存中状态与数据库状态同步）;清除会话清除
  

会话（1st级）缓存，从而释放内存。 所以你需要同时刷新和清除会话以恢复占用的内存。
除此之外，您必须禁用第二级缓存 。否则，即使在清除会话后，所有（或大部分）对象仍然可以访问。



I am trying to process collection of heavy weight elements (images). Size of collection varies between 8000 - 50000 entries. But for some reason after processing 1800-1900 entries my program falls with java.lang.OutOfMemoryError: Java heap space.

In my understanding each time when I call session.getTransaction().commit() program should free heap memory, but looks like it never happens. What do I do wrong? Here is the code:

private static void loadImages( LoadStrategy loadStrategy ) throws IOException {
    log.info( "Loading images for: " + loadStrategy.getPageType() );

    Session session = sessionFactory.openSession();
    session.setFlushMode( FlushMode.COMMIT );
    Query query = session.createQuery( "from PageRaw where pageType = :pageType and pageStatus = :pageStatus and sessionId = 1" );
    query.setString( "pageStatus", PageStatus.SUCCESS.name() );
    query.setString( "pageType", loadStrategy.getPageType().name() );
    query.setMaxResults( 50 );

    List<PageRaw> pages;
    int resultNum = 0;

    do {

        session.getTransaction().begin();

        log.info( "Get pages statring form " + resultNum + " position" );
        query.setFirstResult( resultNum );
        resultNum += 50;
        pages = query.list();
        log.info( "Found " + pages.size() + " pages" );


        for (PageRaw pr : pages ) {
            Set<String> imageUrls = new HashSet<>();
            for ( UrlLocator imageUrlLocator : loadStrategy.getImageUrlLocators() ) {
                imageUrls.addAll(
                        imageUrlLocator.locateUrls( StringConvector.toString( pr.getSourceHtml() ) )
                );
            }

            removeDeletedImageRaws( pr.getImages(), imageUrls );
            loadNewImageRaws( pr.getImages(), imageUrls );
        }

        session.getTransaction().commit();

    } while ( pages.size() > 0 );

    session.close();
}

解决方案

You have confused flushing with clearing:

• flushing a session executes all pending statements against the database (it synchronizes the in-memory state with the database state);

• clearing a session purges the session (1st-level) cache, thus freeing memory.

So you need to both flush and clear a session in order to recover the occupied memory.

In addition to that, you must disable the 2nd-level cache. Otherwise all (or most of) the objects will remain reachable even after clearing the session.

这篇关于在处理大量元素时，休眠内存异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！