MongoDB游标(OutOfMemory)中的内存泄漏? [英] Memory leak in MongoDB cursor (OutOfMemory)?

问题描述

这就是我读取大型MongoDB表的方式(每个对象的属性中都有非常大的数据块):

This is how I'm reading my large MongoDB table (every object has very big chunks of data in its attributes):

DBCursor cursor = collection.find(/* my query */);
while (cursor.hasNext()) {
  DBObject object = cursor.next();
  doSomething(object); // no data stays in memory
}
cursor.close();

我得到了:

java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding$StringDecoder.decode(Unknown Source)
at java.lang.StringCoding.decode(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at org.bson.BasicBSONDecoder$BSONInput.readUTF8String(BasicBSONDecoder.java:463)
at org.bson.BasicBSONDecoder.decodeElement(BasicBSONDecoder.java:155)
at org.bson.BasicBSONDecoder._decode(BasicBSONDecoder.java:79)
at org.bson.BasicBSONDecoder.decode(BasicBSONDecoder.java:57)
at com.mongodb.DefaultDBDecoder.decode(DefaultDBDecoder.java:56)
at com.mongodb.Response.<init>(Response.java:83)
at com.mongodb.DBPort.go(DBPort.java:124)
at com.mongodb.DBPort.call(DBPort.java:74)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:286)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:257)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:310)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:295)
at com.mongodb.DBCursor._check(DBCursor.java:368)
at com.mongodb.DBCursor._hasNext(DBCursor.java:459)
at com.mongodb.DBCursor.hasNext(DBCursor.java:484)

在处理200-300个对象后引发异常.驱动程序是否将数据保存在内存中?我正在使用

Exception is thrown after 200-300 objects are processed. Does the driver holds data in memory? I'm using

<dependency>
  <groupId>org.mongodb</groupId>
  <artifactId>mongo-java-driver</artifactId>
  <version>2.10.1</version>
</dependency>

推荐答案

驱动程序将在内存中保存的对象数量取决于游标的批处理大小.当您未明确设置默认批处理大小时，我不确定默认的批处理大小是多少.根据文档的最大大小和可用的堆空间量，您应该相应地设置批处理大小.请记住，MongoDB当前支持的最大文档大小为16MB.

The number of objects the driver will hold in memory depends on the cursor's batch size. I'm not sure what the default batch size is when you don't set one explicitly. Depending on the maximum size of your documents and the amount of heap space available, you should set the batch size accordingly. Keep in mind that the current maximum document size supported by MongoDB is 16MB.

请参见 DBCursor.batchSize().

这篇关于MongoDB游标(OutOfMemory)中的内存泄漏?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！