如何解决"java.lang.OutOfMemoryError:Java堆空间"? [英] How do I solve a "java.lang.OutOfMemoryError: Java heap space"?
问题描述
我正在编写一些代码，以将非常大的纯文本文件解析为持久存储到数据库中的对象.这正在处理文件的各个部分(即，如果我在"前2000行中居高")，但是在尝试处理整个文件时遇到了java.lang.OutOfMemoryError: Java heap space错误.
I am writing some code to parse a very large flat text file into objects which are persisted to a database. This is working on sections of the file (i.e. if I 'top' the first 2000 lines), but I am running into a java.lang.OutOfMemoryError: Java heap space error when I try and process the full file.
我正在使用BufferedReader逐行读取文件，并且给人的印象是，这否定了将整个文本文件加载到内存中的要求.希望我的代码是不言自明的.我已经通过Eclipse Memory Analyser运行了我的代码，它告诉我:
I am using a BufferedReader to read the file line by line, and I was under the impression that this negates the requirement to load the entire text file into memory. Hopefully my code is fairly self-explanatory. I have run my code through the Eclipse Memory Analyser, which informs me that:
线程java.lang.Thread @ 0x27ee0478 main保留局部变量，其总大小为69,668,888(98.76％)个字节.
内存在由<系统类加载器"> **
The thread java.lang.Thread @ 0x27ee0478 main keeps local variables with total size 69,668,888 (98.76%) bytes.
The memory is accumulated in one instance of "char[]" loaded by "<system class loader>"**
非常感谢有用的评论！
乔纳森
public ArrayList<Statement> parseGMIFile(String filePath)
throws IOException {
ArrayList<Statement> statements = new ArrayList<Statement>();
// Statement Properties
String sAccount = "";
String sOffice = "";
String sFirm = "";
String sDate1 = "";
String sDate2 = "";
Date date = new Date();
StringBuffer sData = new StringBuffer();
BufferedReader in = new BufferedReader(new FileReader(filePath));
String line;
String prevCode = "";
int lineCounter = 1;
int globalLineCounter = 1;
while ((line = in.readLine()) != null) {
// We extract the GMI code from the end of the first line
String newCode = line.substring(GMICODE_START_POS).trim();
// Extract date
if (newCode.equals(prevCode)) {
if (lineCounter == DATE_LINE) {
sDate1 = line.substring(DATE_START_POS, DATE_END_POS).trim();}
if (lineCounter == DATE_LINE2) {
sDate2 = line.substring(DATE_START_POS, DATE_END_POS).trim();}
if (sDate1.equals("")){
sDate1 = sDate2;}
SimpleDateFormat formatter=new SimpleDateFormat("MMM dd, yyyy");
try {
date=formatter.parse(sDate1);
} catch (ParseException e) {
e.printStackTrace();
}
sFirm = line.substring(FIRM_START_POS, FIRM_END_POS);
sOffice = line.substring(OFFICE_START_POS, OFFICE_END_POS);
sAccount = line.substring(ACCOUNT_START_POS,
ACCOUNT_END_POS);
lineCounter++;
globalLineCounter++;
sData.append(line.substring(0, END_OF_DATA)).append("\n");
} else {
// Instantiate New Statement Object
Statement stmt = new Statement(sAccount, sOffice, sFirm,
date, sData.toString());
// Add to collection
statements.add(stmt);
// log.info("-----------NEW STATEMENT--------------");
sData.setLength(0);
lineCounter = 1;
}
prevCode = newCode;
}
return statements;
}
STACKTRACE: Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dbPopulator' defined in class path resource [app-context.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: Java heap space
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1401)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:512)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:450)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:287)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:189)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:557)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:842)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:416)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:93)
at Main.main(Main.java:11)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuffer.append(StringBuffer.java:224)
at services.GMILogParser.parseGMIFile(GMILogParser.java:133)
at services.DBPopulator.init(DBPopulator.java:27)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1529)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1468)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1398)
... 12 more
推荐答案
在启动参数中添加更多内存是恕我直言的错误.这些参数适用于整个应用程序.并且可能会因增加gc次数而受到处罚.此外，您可能事先不知道尺寸.
Adding more memory in the start parameters is IMHO a mistake. Those parameters are application wide. And may penalize by increasing gc times. Moreover, you might not know the size in advance.
您使用MemoryMappedFiles并查看 java.nio.* 这样做.这样一来，您就可以在读取时加载内容，并且内存不会放置在普通的内存空间中.
You use MemoryMappedFiles and look at the java.nio.* to do so. Doing so you can load as you read, and the memory is not placed in the ordinary memory space.
通过低级阅读，您可以按可变长度的块进行阅读.速度很重要.如果您的文件很大，则可能会花费太多时间来读取它.并且，您在JVM中存储的Objects数量使GC正常工作，并且应用程序速度变慢.
从Java参考资料:
By reading at a low level you do it in chunks of variable length. And the speed is important. If your file is large, it may take too much time to read it. And the quantity of Objects you store in JVM makes the GC works and the application slows down.
From the java reference:
-
可以将
byte buffer分配为直接缓冲区，在这种情况下，Java虚拟机将尽最大努力直接在其上执行native I/O operations.
A
byte buffercan be allocated as a direct buffer, in which case the Java virtual machine will make a best effort to performnative I/O operationsdirectly upon it.
可以通过将文件的某个区域直接映射到内存来创建byte buffer，在这种情况下，可以使用MappedByteBuffer类中定义的一些与文件相关的其他操作.
A byte buffer can be created by mapping a region of a file directly into memory, in which case a few additional file-related operations defined in the MappedByteBuffer class are available.
A byte buffer可以按大端或小端字节顺序，以任何非布尔原始类型的二进制数据的异构或同质序列来访问其内容.
A byte buffer provides access to its content as either a heterogeneous or homogeneous sequence of binary data of any non-boolean primitive type, in either big-endian or little-endian byte order.
这篇关于如何解决"java.lang.OutOfMemoryError:Java堆空间"?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
