在磁盘上存储大量不同大小的对象 [英] Storing large numbers of varying size objects on disk

问题描述

我需要开发一个用于存储大量（10到100的数千个）对象的系统。每个对象都是类似电子邮件的 - 有一个主文本正文和几个有限大小的辅助文本字段。身体将从几个字节到几KB大小。

I need to develop a system for storing large numbers (10's to 100's of thousands) of objects. Each object is email-like - there is a main text body, and several ancillary text fields of limited size. A body will be from a few bytes, to several KB in size.

每个项目都有一个唯一的ID（可能是一个GUID）来标识它。

Each item will have a single unique ID (probably a GUID) that identifies it.

只有在添加对象时，才会写入商店。会经常阅读。删除将是罕见的。数据几乎都是人类可读的文本，所以它将很容易压缩。

The store will only be written to when an object is added to it. It will be read often. Deletions will be rare. The data is almost all human readable text so it will be readily compressible.

一个允许我发布I / O并管理内存和缓存的系统将是理想的。

A system that lets me issue the I/Os and mange the memory and caching would be ideal.

我将把索引保存在内存中，使用它将索引映射到对象的单个（和主）键。一旦我有钥匙，那么我将从磁盘或缓存中加载它。

I'm going to keep the indexes in memory, using it to map indexes to the single (and primary) key for the objects. Once I have the key, then I'll load it from disk, or the cache.

数据管理系统需要成为我应用程序的一部分 - 我不想依赖于OS服务。或单独安装包。本机（C ++）将是最好的，但一个管理（C＃）的事情会是好的。

The data management system needs to be part of my application - I do not want to depend on OS services. Or separately installed packages. Native (C++) would be best, but a manged (C#) thing would be ok.

我相信数据库是一个明显的选择，但是这需要超快的查找和加载到对象的内存中。我没有数据库技术的经验，我担心一般关系系统不会有效地处理所有这些可变大小的数据。

I believe that a database is an obvious choice, but this needs to be super-fast for look up and loading into memory of an object. I am not experienced with data base tech and I'm concerned that general relational systems will not handle all this variable sized data efficiently.

（请注意，这与我的工作无关 - 其​​个人项目。）

(Note, this has nothing to do with my job - its a personal project.)

在您的经验，传统关系数据库的可行替代方案是什么？或者DB可以为此工作吗？

In your experience, what are the viable alternatives to a traditional relational DB? Or would a DB work well for this?

推荐答案

查看 SQLite ，它具有许多可用的编程语言和环境的绑定，就像 Berkeley DB ，磁盘上的数据库，而不需要安装数据库引擎。

Look at SQLite, it has bindings for many programming languages and environments available and is, like the Berkeley DB, a database on disk without the need for a database engine installation.

如果只添加正确的索引，查找将非常快，因为它是一个基于集合的数据库，您仍然可以进行批量查询和类似。

If you just add the right indexes, lookups will be very fast, and since it is a set-based database at heart, you can still do bulk queries and similar.

这篇关于在磁盘上存储大量不同大小的对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！