如何在JVM运行之间持久保留大型Java对象 [英] How to persist large Java object between invocations of JVM run

问题描述

我正在处理以二进制格式存储在磁盘上的大型图形.从磁盘(SSD)读取图形并构建图形大约需要一个小时.一旦构建，图就永远不会改变.该图大约需要50GB的内存，这对于服务器来说不是问题.但是，我们经常要在图形上进行大量实验，并且要花大量时间来加载图形.我想知道是否有任何方法可以将对象保留在内存中，以便JVM可以从本质上将对象定位在内存中.

I'm working with large graphs that are stored on disk in a binary format. Reading the graph from disk (SSD) and constructing the graph takes roughly an hour. Once constructed the graph never changes. The graph takes roughly 50GB of memory, which isn't a problem for the server. However, we often want to do lots of experiments on the graph and paying the hour graph-loading time gets expensive. I'm wondering if there is any way to persist the object in memory so that the JVM can essentially locate the object in memory.

我知道JVM在进程之间具有内存共享，但是我看不到任何可以让您共享整个对象而无需将对象序列化为字节的东西(考虑到昂贵的重建工作，这样做可能会很慢).由于对象很大(50GB)，数据库解决方案似乎也很慢.由于我们没有修改对象(它实际上是静态的)，因此我不必担心进程之间的并发问题.

I know the JVM has memory sharing between processes, but I haven't see anything that lets you share whole object without serializing the object to bytes (which would likely be slow given the expensive reconstruction). Database solutions also seem slow because of the bulk of the object (50GB). Since we aren't modifying the object (it's effectively static), I'm not concerned about concurrency issues between processes.

我所见过的最好的主意是使用

The best idea I've seen is to use a FileChannel to map the serialized object into memory using an always persistent JVM and then have the second JVM read from that FileChannel to deserialize the object. Any other suggestions would be much appreciated!

推荐答案

我建议使用 ChronicleMa p(由我帮助设计)

I suggest using ChronicleMap (which I helped design)

是:

• 坚持
• 共享
• 堆外
• 可以大于主内存
• 具有使序列化成本最小化的选项.

例如 https://github.com/OpenHFT/Chronicle-Map/blob/master/docs/CM_Tutorial.adoc

interface PostalCodeRange {
    int minCode();
    void minCode(int minCode);

    int maxCode();
    void maxCode(int maxCode);
}

ChronicleMap<Integer, PostalCodeRange> cityPostalCodes = ChronicleMap
    .of(CharSequence.class, PostalCodeRange.class)
    .averageKey("Amsterdam")
    .entries(50_000)
    .createOrRecoverPersistedTo(cityPostalCodesFile, false);

注意:在这种情况下，该值是堆外内存上的动态重量，它使您可以在不反序列化对象的情况下访问字段.

NOTE: The value in this cases is a flyweight over off heap memory which allows you to access fields without deserializaing an object.

这篇关于如何在JVM运行之间持久保留大型Java对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！