Neo4j OutOfMemory 问题 [英] Neo4j OutOfMemory problem
问题描述
这是我的 Main.java 源代码.它是从 neo4j-apoc-1.0 示例中抓取的.修改目标为存储2个节点1个关系的1M条记录:
This is my source code of Main.java. It was grabbed from neo4j-apoc-1.0 examples. The goal of modification to store 1M records of 2 nodes and 1 relation:
package javaapplication2;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.RelationshipType;
import org.neo4j.graphdb.Transaction;
import org.neo4j.kernel.EmbeddedGraphDatabase;
public class Main
{
private static final String DB_PATH = "neo4j-store-1M";
private static final String NAME_KEY = "name";
private static enum ExampleRelationshipTypes implements RelationshipType
{
EXAMPLE
}
public static void main(String[] args)
{
GraphDatabaseService graphDb = null;
try
{
System.out.println( "Init database..." );
graphDb = new EmbeddedGraphDatabase( DB_PATH );
registerShutdownHook( graphDb );
System.out.println( "Start of creating database..." );
int valIndex = 0;
for(int i=0; i<1000; ++i)
{
for(int j=0; j<1000; ++j)
{
Transaction tx = graphDb.beginTx();
try
{
Node firstNode = graphDb.createNode();
firstNode.setProperty( NAME_KEY, "Hello" + valIndex );
Node secondNode = graphDb.createNode();
secondNode.setProperty( NAME_KEY, "World" + valIndex );
firstNode.createRelationshipTo(
secondNode, ExampleRelationshipTypes.EXAMPLE );
tx.success();
++valIndex;
}
finally
{
tx.finish();
}
}
}
System.out.println("Ok, client processing finished!");
}
finally
{
System.out.println( "Shutting down database ..." );
graphDb.shutdown();
}
}
private static void registerShutdownHook( final GraphDatabaseService graphDb )
{
// Registers a shutdown hook for the Neo4j instance so that it
// shuts down nicely when the VM exits (even if you "Ctrl-C" the
// running example before it's completed)
Runtime.getRuntime().addShutdownHook( new Thread()
{
@Override
public void run()
{
graphDb.shutdown();
}
} );
}
}
经过几次迭代(大约 150K)后，我收到错误消息:
After a few iterations (around 150K) I got error message:
"java.lang.OutOfMemoryError: Java 堆空间在 java.nio.HeapByteBuffer.(HeapByteBuffer.java:39)在 java.nio.ByteBuffer.allocate(ByteBuffer.java:312)在 org.neo4j.kernel.impl.nioneo.store.PlainPersistenceWindow.(PlainPersistenceWindow.java:30)在 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.allocateNewWindow(PersistenceWindowPool.java:534)在 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.refreshBricks(PersistenceWindowPool.java:430)在 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:122)在 org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:459)在 org.neo4j.kernel.impl.nioneo.store.AbstractDynamicStore.updateRecord(AbstractDynamicStore.java:240)在 org.neo4j.kernel.impl.nioneo.store.PropertyStore.updateRecord(PropertyStore.java:209)在 org.neo4j.kernel.impl.nioneo.xa.Command$PropertyCommand.execute(Command.java:513)在 org.neo4j.kernel.impl.nioneo.xa.NeoTransaction.doCommit(NeoTransaction.java:443)在 org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:316)在 org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:399)在 org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64)在 org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:514)在 org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:571)在 org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:543)在 org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:102)在 org.neo4j.kernel.EmbeddedGraphDbImpl$TransactionImpl.finish(EmbeddedGraphDbImpl.java:329)在 javaapplication2.Main.main(Main.java:62)28.05.2010 9:52:14 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool logWarn警告:[neo4j-store-1M eostore.propertystore.db.strings] 无法分配直接缓冲区"
"java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.(HeapByteBuffer.java:39) at java.nio.ByteBuffer.allocate(ByteBuffer.java:312) at org.neo4j.kernel.impl.nioneo.store.PlainPersistenceWindow.(PlainPersistenceWindow.java:30) at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.allocateNewWindow(PersistenceWindowPool.java:534) at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.refreshBricks(PersistenceWindowPool.java:430) at org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:122) at org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:459) at org.neo4j.kernel.impl.nioneo.store.AbstractDynamicStore.updateRecord(AbstractDynamicStore.java:240) at org.neo4j.kernel.impl.nioneo.store.PropertyStore.updateRecord(PropertyStore.java:209) at org.neo4j.kernel.impl.nioneo.xa.Command$PropertyCommand.execute(Command.java:513) at org.neo4j.kernel.impl.nioneo.xa.NeoTransaction.doCommit(NeoTransaction.java:443) at org.neo4j.kernel.impl.transaction.xaframework.XaTransaction.commit(XaTransaction.java:316) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceManager.commit(XaResourceManager.java:399) at org.neo4j.kernel.impl.transaction.xaframework.XaResourceHelpImpl.commit(XaResourceHelpImpl.java:64) at org.neo4j.kernel.impl.transaction.TransactionImpl.doCommit(TransactionImpl.java:514) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:571) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:543) at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:102) at org.neo4j.kernel.EmbeddedGraphDbImpl$TransactionImpl.finish(EmbeddedGraphDbImpl.java:329) at javaapplication2.Main.main(Main.java:62) 28.05.2010 9:52:14 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool logWarn WARNING: [neo4j-store-1M eostore.propertystore.db.strings] Unable to allocate direct buffer"
伙计们！帮帮我，我做错了什么，我该如何修复?在平台 Windows XP 32 位 SP3 上测试.也许在创建自定义配置中的解决方案?
Guys! Help me plzzz, what I did wrong, how can I repair it? Tested on platform Windows XP 32bit SP3. Maybe solution within creation custom configuration?
thnx 4 每条建议！
thnx 4 every advice!
推荐答案
这是 Windows 上的配置问题，其中 Neo4j 无法使用内存映射缓冲区.相反，在堆上创建了一个 Java 缓冲区.在 1.0 中，这个缓冲区默认为 470MB，比 Windows JVM 的默认堆要多.您有两个选择:
this is a configuration issue on Windows, where Neo4j cannot use memory mapped buffers. Instead, a Java Buffer on the heap is created. In 1.0 this buffer was 470MB per default, which is more than the default heap for the Windows JVM. You have two options:
在具有自动配置的 pom.xml 中切换到 APOC 1.1-SNAPSHOT 而不是 1.0，将最多 50% 的可用 JVM 堆分配给 Neo4j
Switch to APOC 1.1-SNAPSHOT instead of 1.0 in your pom.xml which has an autoconfiguration, assigning max 50% of the available JVM heap to Neo4j
通过运行 Java 将 JVM 堆调整到更大(例如 512Mb)
Adjust the JVM heap to more (e.g. 512Mb) by running Java with
java -Xmx512m ....
java -Xmx512m ....
您甚至可以将其插入到 Eclipse 中运行配置的 JVM 参数下
You can even insert that under JVM arguments in the Run Configurations in Eclipse
如果这有帮助，请告诉我们！
Let us know if this helps!
另外，为每个节点对做一个完整的交易需要很长时间.尝试在第一个循环中打开一个事务，并且每 1000 个节点对只提交一次?
Also, doing a full transaction for every node pair is going to take a long time. Try opening a transaction in the first loop and do commits only every 1000 node pairs?
/彼得
这篇关于Neo4j OutOfMemory 问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
