Java：大型持久散列结构？ [英] Java: large persistent hash structure?

问题描述

我正在java中寻找一个持久散列结构，这是一个简单的键值存储，其中key是唯一的字符串，value是int。每次将现有密钥添加到商店时，密钥的值都会增加。

我需要这个值非常大 - 可能是500万-1亿个密钥。我一直在评估东京内阁 http://fallabs.com/tokyocabinet/javadoc/ ，但没有确定它会如何扩展 - 随着哈希增长，插入时间似乎会变得越来越长。

关于什么可能合适吗？

谢谢

编辑：为了减少磁盘I / O，我将缓存数据内存中的HashMap，然后在缓存增长到一定大小时一次更新持久哈希。

Edit2：其中一个原因对于持久性而言，我的内存有限，4GB，所以我无法适应大的内存结构。

事情Megamap是你在找什么： http://megamap.sourceforge.net/ 。以下是Megamap的主页简介：
$ b

MegaMap是一个
映射（或散列表）的Java实现，可以存储
无限量的数据，仅限可用磁盘空间量
。
映射中存储的对象是
，它们保存在磁盘中。内存高速缓存实现的性能好
。对于所有实际的
原因，
MegaMap可以被认为是具有无限存储空间
的映射
实现。

I'm looking for a persistent hash structure in java, a simple key-value store, where key is a unique string and value is an int. The value of a key is to be incremented each time an existing key is added to the store.

I need this to be quite large - possibly 500m - 1bn keys. I've been evaluating tokyo-cabinet http://fallabs.com/tokyocabinet/javadoc/ but not sure how well it will scale - insert times seem to be getting longer as the hash grows.

Any ideas on what might be appropriate?

Thanks

Edit: In order to reduce disk I/O I'm going to be caching data in an in-memory HashMap, then updating the persistent hash in one go when the cache grows to a certain size.

Edit2: One of the reasons for the persistence is that I have limited RAM, 4GB, so I can't fit a big struture into memory.

解决方案

I thing Megamap is what you are looking for: http://megamap.sourceforge.net/. Here is a short description of Megamap from its homepage:

MegaMap is a Java implementation of a map (or hashtable) that can store an unbounded amount of data, limited only by the amount of disk space available. Objects stored in the map are persisted to disk. Good performance is achieved by an in-memory cache. The MegaMap can, for all practical reasons, be thought of as a map implementation with unlimited storage space.

这篇关于Java：大型持久散列结构？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！