有什么像Redis DB，但不限于RAM大小？ [英] Is there something like Redis DB, but not limited with RAM size?
问题描述
我在寻找符合以下条件的数据库：
- 可以是非持久性的;
- 几乎所有的密钥都需要在3-6小时内更新一次（100M +密钥，总大小为100Gb）
- 能够通过键（或主键）快速选择数据
- 这需要是DBMS（因此LevelDB不合适）
- 数据写入时，数据库群集必须能够提供查询
- 不会在内存中 - 我们的数据集将超出RAM限制
- 水平缩放和复制
- 支持完全重写所有数据（MongoDB在删除数据后不会清除空间）
- C＃和Java支持
这是我使用这样的数据库的过程：
我们有一个分析集群，每4-6小时生成100M记录（50GB）的数据。数据是一个key - array [20]。此数据需要通过前端系统以每秒1-10k请求的速率分发给用户。平均而言，只有〜15％的数据被请求，其余的将在4-6小时内被重写，当下一个数据集生成。
尝试：
- MongoDB。 Datastorage开销，高碎片整理成本。
- Redis。看起来很完美，但是它的内存有限，我们的数据超过了它。
问题是：有什么像Redis，受限于RAM大小？
是的，Redis有两个备选项，不受RAM大小限制，兼容Redis协议：
Ardb（C ++），复制（主 - 从/主 - 主）： https://github.com/yinqiwen/ardb
一个redis协议兼容的永久存储服务器，支持
LevelDB / KyotoCabinet / LMDB作为存储引擎。
Edis ： http://inaka.github.io/edis/
Edis是一个协议兼容的服务器替代Redis，用
Erlang编写。当
持久化比将数据集保存在内存中更重要时，Edis的目标是成为Redis的替代。 Edis
（目前）使用Google的leveldb作为后端。
为了完整性，数据库：
Hyperdex（字符串，整数，浮点数，列表，集合，地图）： http://hyperdex.org/doc/latest/DataTypes/#chap:data-types
快速：HyperDex具有更低的延迟，更高的吞吐量，以及更高的吞吐量，更高的吞吐量和更低的吞吐量。比其他键值商店降低
的差异。
，系统中将添加更多计算机。
线性化。因此，读操作总是返回
插入到系统中的最新值。不只是最终，而是
立即和总是。
在多台计算机上复制数据，因此并发故障（由应用程序确定的限制增加
）不会导致数据丢失。
可搜索：
属性。
脚本和母语提供API。
，并且只需很少的用户维护。
I'm looking for a database matching these criteria:
- May be non-persistent;
- Almost all keys of DB need to be updated once in 3-6 hours (100M+ keys with total size of 100Gb)
- Ability to quickly select data by key (or Primary Key)
- This needs to be a DBMS (so LevelDB doesn't fit)
- When data is written, DB cluster must be able to serve queries (single nodes can be blocked though)
- Not in-memory – our dataset will exceed the RAM limits
- Horizontal scaling and replication
- Support full rewrite of all data (MongoDB doesn't clear space after deleting data)
- C# and Java support
Here's my process of working with such database: We've got an analytics cluster that produces 100M records (50GB) of data every 4-6 hours. The data is a "key - array[20]". This data needs to be distributed to users through a front-end system with a rate of 1-10k requests per second. In average, only ~15% of the data is requested, the rest of it will be rewritten in 4-6 hours when the next data set is generated.
What i tried:
- MongoDB. Datastorage overhead, high defragmentation costs.
- Redis. Looks perfect, but it's limited with RAM and our data exceeds it.
So the question is: is there anything like Redis, but not limited with RAM size?
Yes, there are two alternatives to Redis that are not limited by RAM size while remaining compatible with Redis protocol:
Ardb (C++), replication(Master-Slave/Master-Master): https://github.com/yinqiwen/ardb
A redis-protocol compatible persistent storage server, support LevelDB/KyotoCabinet/LMDB as storage engine.
Edis (Erlang): http://inaka.github.io/edis/
Edis is a protocol-compatible Server replacement for Redis, written in Erlang. Edis's goal is to be a drop-in replacement for Redis when persistence is more important than holding the dataset in-memory. Edis (currently) uses Google's leveldb as a backend.
And for completeness here is another data-structures database:
Hyperdex (Strings, Integers, Floats, Lists, Sets, Maps): http://hyperdex.org/doc/latest/DataTypes/#chap:data-types
HyperDex is:
- Fast: HyperDex has lower latency, higher throughput, and lower variance than other key-value stores.
- Scalable: HyperDex scales as more machines are added to the system.
- Consistent: HyperDex guarantees linearizability for key-based operations. Thus, a read always returns the latest value inserted into the system. Not just "eventually," but immediately and always.
- Fault Tolerant: HyperDex automatically replicates data on multiple machines so that concurrent failures, up to an application-determined limit, will not cause data loss. Searchable:
- HyperDex enables efficient lookups of secondary data attributes.
- Easy-to-Use: HyperDex provides APIs for a variety of scripting and native languages.
- Self-Maintaining: A HyperDex is self-maintaining and requires little user maintenance.
这篇关于有什么像Redis DB，但不限于RAM大小？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
