Java Neo4J内存不足 [英] Java Neo4J out of memory
本文介绍了Java Neo4J内存不足的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
这有点像: Neo4j OutOfMemory问题
但是它已经过时了，据我所知，解决方案显然也是如此.
因此，我试图插入约5.5万个关系的100K个节点(实际上是削减了数据集，所以现在它更像是<2.8万个关系的100K个节点).
过一会儿，它的内存不足了，我得到了一个类似这样的异常:
Exception in thread "GC-Monitor" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at java.lang.StringBuilder.toString(Unknown Source)
at org.neo4j.kernel.impl.util.StringLogger$ActualStringLogger.logMessage(StringLogger.java:276)
at org.neo4j.kernel.impl.cache.MeasureDoNothing.run(MeasureDoNothing.java:85)
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.addBefore(Unknown Source)
at java.util.LinkedList.add(Unknown Source)
at org.neo4j.kernel.impl.nioneo.store.IdGeneratorImpl.freeId(IdGeneratorImpl.java:291)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.freeId(CommonAbstractStore.java:382)
at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doRollback(WriteTransaction.java:315)
at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.rollback(XaTransaction.java:278)
at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.rollback(XaResourceManager.java:518)
at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.rollback(XaResourceHelpImpl.java:111)
at org.neo4j.kernel.impl.transaction.TransactionImpl.doRollback(TransactionImpl.java:558)
at org.neo4j.kernel.impl.transaction.TxManager.rollback(TxManager.java:610)
at org.neo4j.kernel.impl.transaction.TransactionImpl.rollback(TransactionImpl.java:129)
at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:119)
at sqlToGraph.SqlToGraph.main(SqlToGraph.java:81)
我尝试将-Xmx1500m传递给java，这大约是我可以传递的限制，因为在它抱怨不能分配堆空间之前，它已经通过了.它的持续时间明显更长，但仍然无法结束.
以下是(略作编辑的)代码:
/* Postgres query and setup stuff cut */
Transaction tx = graphDb.beginTx();
try {
while (rs.next()) {
user_lo = rs.getInt(1);
user_hi = rs.getInt(2);
n_lo = getOrCreate(user_lo, graphDb);
n_lo.setProperty("user_id", user_lo);
n_lo.setProperty("total", rs.getInt(3));
n_hi = getOrCreate(user_ graphDb);
n_hi.setProperty("user_id", user_hi);
n_hi.setProperty("total", rs.getInt(4));
relationship = n_lo.createRelationshipTo(n_ RelTypes.PLAYED_WITH);
relationship.setProperty("mean_percent", rs.getDouble(5));
}
tx.success();
} finally {
tx.finish();
}
graphDb.shutdown();
解决方案
在此处添加另一个答案.因此，考虑到代码，问题在于您永远不会提交事务.事务数据一直保存在内存中，直到提交为止，因此所有创建的节点和关系都将一直位于内存中等待提交，这就是为什么最终获得OOM的原因.
我建议更改此代码:
/* Postgres query and setup stuff cut */
Transaction tx = graphDb.beginTx();
try {
for (int i = 0; rs.next(); i++) {
user_lo = rs.getInt(1);
user_hi = rs.getInt(2);
n_lo = getOrCreate(user_lo, graphDb);
n_lo.setProperty("user_id", user_lo);
n_lo.setProperty("total", rs.getInt(3));
n_hi = getOrCreate(user_ graphDb);
n_hi.setProperty("user_id", user_hi);
n_hi.setProperty("total", rs.getInt(4));
relationship = n_lo.createRelationshipTo(n_ RelTypes.PLAYED_WITH);
relationship.setProperty("mean_percent", rs.getDouble(5));
// Commit every now and then to free memory.
if ( i > 0 && i % 10000 == 0 ) {
tx.success();
tx.finish();
tx = graphDb.beginTx();
}
}
tx.success();
} finally {
tx.finish();
}
graphDb.shutdown();
This is a bit like this: Neo4j OutOfMemory problem
But it's outdated and apparently so are the solutions as far as I can tell.
So I'm trying to insert around 100K nodes with 5.5M relations (I actually cut down my data set so it's now more like <100K nodes with 2.8M relations).
After a while, it runs out of memory and I get an exception like so:
Exception in thread "GC-Monitor" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at java.lang.StringBuilder.toString(Unknown Source)
at org.neo4j.kernel.impl.util.StringLogger$ActualStringLogger.logMessage(StringLogger.java:276)
at org.neo4j.kernel.impl.cache.MeasureDoNothing.run(MeasureDoNothing.java:85)
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedList.addBefore(Unknown Source)
at java.util.LinkedList.add(Unknown Source)
at org.neo4j.kernel.impl.nioneo.store.IdGeneratorImpl.freeId(IdGeneratorImpl.java:291)
at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.freeId(CommonAbstractStore.java:382)
at org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.doRollback(WriteTransaction.java:315)
at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.rollback(XaTransaction.java:278)
at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.rollback(XaResourceManager.java:518)
at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.rollback(XaResourceHelpImpl.java:111)
at org.neo4j.kernel.impl.transaction.TransactionImpl.doRollback(TransactionImpl.java:558)
at org.neo4j.kernel.impl.transaction.TxManager.rollback(TxManager.java:610)
at org.neo4j.kernel.impl.transaction.TransactionImpl.rollback(TransactionImpl.java:129)
at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:119)
at sqlToGraph.SqlToGraph.main(SqlToGraph.java:81)
I've tried passing -Xmx1500m to java, which is about the limit of what I can pass because before it complains about not being able to allocate the heap space. It lasts significantly longer, but still doesn't finish.
Here is the (slightly edited) code:
/* Postgres query and setup stuff cut */
Transaction tx = graphDb.beginTx();
try {
while (rs.next()) {
user_lo = rs.getInt(1);
user_hi = rs.getInt(2);
n_lo = getOrCreate(user_lo, graphDb);
n_lo.setProperty("user_id", user_lo);
n_lo.setProperty("total", rs.getInt(3));
n_hi = getOrCreate(user_hi, graphDb);
n_hi.setProperty("user_id", user_hi);
n_hi.setProperty("total", rs.getInt(4));
relationship = n_lo.createRelationshipTo(n_hi, RelTypes.PLAYED_WITH);
relationship.setProperty("mean_percent", rs.getDouble(5));
}
tx.success();
} finally {
tx.finish();
}
graphDb.shutdown();
解决方案
Adding another answer here. So given the code the problem is that you never commit your transaction. Transaction data is kept in memory until committed so all your created nodes and relationships will just sit in memory awaiting a commit and that's why you eventually get OOM.
I'd propose this code change:
/* Postgres query and setup stuff cut */
Transaction tx = graphDb.beginTx();
try {
for (int i = 0; rs.next(); i++) {
user_lo = rs.getInt(1);
user_hi = rs.getInt(2);
n_lo = getOrCreate(user_lo, graphDb);
n_lo.setProperty("user_id", user_lo);
n_lo.setProperty("total", rs.getInt(3));
n_hi = getOrCreate(user_hi, graphDb);
n_hi.setProperty("user_id", user_hi);
n_hi.setProperty("total", rs.getInt(4));
relationship = n_lo.createRelationshipTo(n_hi, RelTypes.PLAYED_WITH);
relationship.setProperty("mean_percent", rs.getDouble(5));
// Commit every now and then to free memory.
if ( i > 0 && i % 10000 == 0 ) {
tx.success();
tx.finish();
tx = graphDb.beginTx();
}
}
tx.success();
} finally {
tx.finish();
}
graphDb.shutdown();
这篇关于Java Neo4J内存不足的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
