数据库连接和OutOfMemoryError：Java堆空间 [英] Database connections and OutOfMemoryError: Java Heap Space

问题描述

去年夏天，我做了一个Java应用程序，它将解析一些PDF文件并获取它们包含的信息，将它们存储在SQLite数据库中。

一切都很好，我每周都在数据库中添加新文件，没有任何问题。

现在，我试图提高我的应用程序的速度，我想看看如何如果我在一个新的数据库中解析了我过去两年的所有文件。这是我开始得到这个错误： OutOfMemoryError：Java堆空间。我以前没有得到它，因为我只是解析约25个新文件每周，但似乎解析1000多个文件一个接一个是更多的要求。

我部分解决了问题：我确保在每次调用数据库后关闭我的连接，错误消失了，但是成本巨大。解析文件现在是不可能的慢。对于我的ResultSet和语句/ PreparedStatements，我已经在每次调用后关闭它们。

我想有一些我不知道当我应该关闭我的连接和当我应该继续重复使用相同的。我认为，因为自动提交开启，它提交后，每个事务（选择，更新，插入等）和连接释放额外的内存，它使用。我可能是错误的，因为当我解析太多的文件，我最终得到我提到的错误。

一个简单的解决方案是关闭它后每x调用，但后来我不明白为什么，我可能会得到相同的错误后。任何人都可以解释什么时候我应该关闭我的连接（如果除了当我完成）。如果我只是应该做，当我完成，那么有人可以解释我应该避免这个错误吗？

顺便说一下，我didn因为我在我的在线MySQL数据库上运行我的程序时遇到了同样的错误。

编辑
正如Deco和Mavrav指出的，也许问题不是我的Connection。也许是文件，所以我要发布我使用的代码调用函数来逐个解析文件：

  public static void visitAllDirsAndFiles（File dir）{
 if（dir.isDirectory（））{
 String [] children = dir.list 
 
 for（int i = 0; i  visitAllDirsAndFiles（new File（dir，children [i]））; 
} 
} 
 else {
 try {
 // System.out.println（File：+ dir）; 
 BowlingFilesReader.readFile（dir，playersDatabase）; 
} 
 catch（Exception exc）{
 System.out.println（文件中的其他异常：+ dir）; 
} 
} 
} 
  

使用目录，它使用我刚刚创建的File对象递归调用该函数。我的方法然后检测到它是一个文件并调用 BowlingFilesReader.readFile（dir，playersDatabase）;

解决方案

您对开放结果集和连接的第一本能是好的，虽然也许不完全是原因。让我们先从您的数据库连接开始。

数据库

尝试使用数据库连接池库Apache Commons DBCP（BasicDataSource是一个很好的起点）： http://commons.apache.org/dbcp/
您仍然需要关闭数据库对象，但这将使数据库前端平稳运行。

---

JVM内存

增加给JVM的内存大小。您可以通过添加-Xmx和之后的内存量来执行此操作，例如：

• -Xmx64m < - 这将使JVM 64

你的数字，但是，在JVM上投入更多的内存不会解决内存泄漏。您可以使用JConsole或JVisualVM（包含在您的JDK的bin /文件夹中）来观察您使用的内存量。




线程





假设您正在执行的用于解析这些记录的操作是可以线程化的，您可以通过将它们分开来提高操作的速度。





希望这会有帮助。




Last summer, I made a Java application that would parse some PDF files and get the information they contain to store them in a SQLite database.

Everything was fine and I kept adding new files to the database every week or so without any problems.

Now, I'm trying to improve my application's speed and I wanted to see how it would fare if I parsed all the files I have from the last two years in a new database. That's when I started getting this error: OutOfMemoryError: Java Heap Space. I didn't get it before because I was only parsing about 25 new files per week, but it seems like parsing 1000+ files one after the other is a lot more demanding.

I partially solved the problem: I made sure to close my connection after every call to the database and the error went away, but at a huge cost. Parsing the files is now unbearably slow. As for my ResultSets and Statements / PreparedStatements, I'm already closing them after every call.

I guess there's something I don't understand about when I should close my connection and when I should keep re-using the same one. I thought that since auto-commit is on, it commits after every transaction (select, update, insert, etc.) and the connection releases the extra memory it was using. I'm probably wrong since when I parse too many files, I end up getting the error I'm mentioning.

An easy solution would be to close it after every x calls, but then again I won't understand why and I'm probably going to get the same error later on. Can anyone explain when I should be closing my connections (if at all except when I'm done)? If I'm only supposed to do it when I'm done, then can someone explain how I'm supposed to avoid this error?

By the way, I didn't tag this as SQLite because I got the same error when I tried running my program on my online MySQL database.

Edit As it has been pointed out by Deco and Mavrav, maybe the problem isn't my Connection. Maybe it's the files, so I'm going to post the code I use to call the function to parse the files one by one:

public static void visitAllDirsAndFiles(File dir){
    if (dir.isDirectory()){
        String[] children = dir.list();

        for (int i = 0; i < children.length; i++){
            visitAllDirsAndFiles(new File(dir, children[i]));
        }
    }
    else{
        try{
            // System.out.println("File: " + dir);
            BowlingFilesReader.readFile(dir, playersDatabase);
        }
        catch (Exception exc){
            System.out.println("Other exception in file: " + dir);
        }
    }
}

So if I call the method using a directory, it recursively calls the function again using the File object I just created. My method then detects that it's a file and calls BowlingFilesReader.readFile(dir, playersDatabase);

The memory should be released when the method is done I think?

解决方案

Your first instinct on open resultsets and connections was good, though maybe not entirely the cause. Let's start with your database connection first.

Database

Try using a database connection pooling library, such as the Apache Commons DBCP (BasicDataSource is a good place to start): http://commons.apache.org/dbcp/ You will still need to close your database objects, but this will keep things running smoothly on the database front.

---

JVM Memory

Increase the size of the memory you give to the JVM. You may do so by adding -Xmx and a memory amount after, such as:

• -Xmx64m <- this would give the JVM 64 megs of memory to play with
• -Xmx512m <- 512 megs

Be careful with your numbers, though, throwing more memory at the JVM will not fix memory leaks. You may use something like JConsole or JVisualVM (included in your JDK's bin/ folder) to observe how much memory you are using.

Threading

You may increase the speed of your operations by threading them out, assuming the operation you are performing to parse these records is threadable. But more information might be necessary to answer that question.

Hope this helps.

这篇关于数据库连接和OutOfMemoryError：Java堆空间的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！