假脱机到磁盘的Java集合 [英] Java collections that spool to disk

问题描述

由于在事务过程中构建了一些非常大的列表和集合，并且在事务结束时只迭代了一次，因此我在Java中耗尽内存。是否有任何提供Java集合的库可以在集合大小超过给定阈值时将其可序列化内容假脱机？

I'm running out of memory in Java due to some very large Lists and Sets built up over the course of a transaction and iterated over just once at transaction's end. Are there any libraries that provide Java collections that can spool their serializable contents to disk when the collection size exceeds a given threshold?

推荐答案

我不会发布你的示例代码，因为它会变得太长，但这就是我以前做过的方式：

I won't post you example code because it would become too long, but this is how I done it before:

1. 扩展 LinkedBlockingQueue 。


2. 覆盖其商品， put ， poll ， take 和 remove 方法。 示例：如果超类'商品返回false（达到容量），然后我将开始序列化到磁盘。


3. 同样，在执行实现中，检查内存中是否有任何现有元素，如果没有，然后你开始从磁盘读取（并删除第一个读取记录，因为它现在将驻留在内存中;或者你可以批量读取记录）。


4. 分配每个实例这样一个队列的文件系统安全标识符，以便我可以使用它为它创建文件系统安全的文件名。另外，为了更进一步，我可能会使用当前用户的主目录作为这些队列序列化到磁盘上的位置。

1. Extend LinkedBlockingQueue.
2. Override its offer, put, poll, take and remove methods. Example: if superclass' offer returns false (capacity reached), then I would start serializing to disk.
3. Likewise, in take implementation, you check if you have any present elements in memory at all, and, if not, then you start reading from disk (and removing the first record read because it will now reside in memory; or you can read records in batches, of course).
4. Assign each instance of such a queue a filesystem-safe identifier, so that I can use it to create filesystem-safe file name for it. Also, to take it further, I would probably use current user's home directory as a place for these queues to be serialized onto disk.

这样，99％的磁盘序列化队列已准备就绪，您只需将额外的功能放在正确的位置。您需要彻底阅读Java的 BlockingQueue 界面的文档，但是，这是值得的，因为您只需要添加一些额外的功能，而不是写整件事从头开始。

That way, 99% of your disk-serialized queue is ready, and you only put your extra functionality at exactly the right spots. You would need to thoroughly read the documentation of Java's BlockingQueue interface, but hey, it's worth your time because you will only add that little extra bit of functionality you need, rather than writing the whole thing from scratch.

希望这会有所帮助。

这篇关于假脱机到磁盘的Java集合的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！