如何在Delphi中实现一个巨大但简单的索引StringList？ [英] How Should I Implement a Huge but Simple Indexed StringList in Delphi?

问题描述

我使用Delphi 2009.我有一个非常简单的数据结构，有两个字段：

I am using Delphi 2009. I have a very simple data structure, with 2 fields:

1. 我需要检索，通常长度为4到15个字符。


2. 一个字符串，数据字段可以是任意大小，从1个字符到10,000个字符。 li>
   

1. A string that is the key field I need to retrieve by and is usually 4 to 15 characters in length.
2. A string that is the data field that may be any size, from 1 character up to say 10,000 characters.

难度在于我可能有几百万条记录，所以它们的总大小可能超过10 GB。显然，我正在寻找磁盘上的解决方案而不是内存中的解决方案。

The difficulty is that I may have several million of these records, so they may total as much or more than 10 GB in size. Obviously, I'm looking for an on-disk solution rather than an in-memory solution.

我的程序需要根据键字段随机检索这些记录。这就是需要尽可能高效的部分。

My program needs to randomly retrieve these records, based on the key field. That's that part that needs to be made as efficient as possible.

我应该使用数据库这样一个简单的结构，如果是这样，哪个数据库最好处理这是最简单的实现？

Should I use a database for such a simple structure, and if so, which database would be best to handle this and be simplest to implement?

或者，是否有一个简单的磁盘数据结构，不需要一个完整的数据库，也可以工作吗？

Alternatively, is there a simple on-disk data structure not requiring a full-blown database that would work just as well?

好吧，我需要的只是一个答案，让我回到现实。我正在寻找比简单的数据库更简单的东西。但是当no-duh的答案是使用一个数据库，然后我意识到我已经回答了这个问题与我自己的答案另一个问题：

Well, all I needed was the one answer to kick me back into reality. I was looking for something simpler than even a simple database. But when the no-duh answer is to use a database, then I realize I've already answered this question with my own answer to another question: Best database for small applications and tools.

我的答案是 http://www.yunqa.de/delphi/doku.php/products/sqlite3/rel =nofollow noreferrer> DISQLite3 我指定的原因。

这是一些更好的答案。那很棒。我可以尝试几种不同的方法来查看效果最好的方法。

A few more good answers with some possibilities. That's great. I'll be able to try a few different methods to see what works best.

更多的沉思，更改接受的答案GpStructuredStorage解决方案。

More contemplation, and I have had to change the accepted answer to the GpStructuredStorage solution.

在我的情况下，一百万条记录总共几个千兆字节将给数据库结构带来压力。特别地，用于在大多数数据库中存储索引的B *树是快速的，但对于某些操作（例如重新索引一百万个值）将减慢。

In my case, a million records totalling several Gigabytes will put a strain on a database structure. Specifically, the B* tree that is used to store the index in most databases is fast, but will slow down for some operations such as reindexing a million values.

对于一个索引，你发现比B *快的唯一的东西是哈希表。这正是gabr建议的GpStructuredStorage解决方案中提供的内容。我认为这是非常优雅的方式，他分割哈希值以提供一个4级目录结构。

The only thing you'll find faster than B* for an index is a hash table. And that is precisely what is provided in gabr's suggested addition to the GpStructuredStorage solution. I think it's quite elegant the way he segmented the hash value to give a 4 level directory structure.

我可以去哈希解决方案的关键原因是，我只需要通过密钥进行随机访问。我不需要按键排序。如果需要排序，那么哈希表的速度的增益将会丢失，数据库系统将是无脑的赢家。

The key reason why I can go to a hash solution is that I only need random access by the keys. I do not need to sort by the keys. If sorting was needed, then the gains of the speed of the hash table would be lost and the database system would be a no-brain winner.

当我下到实现这一点，我应该做一个比较这种技术与数据库。也许我会比较Firebird和SQLite这两个都值得对手。

When I get down to implementing this, I should do a comparison of this technique versus a database. Maybe I'll compare with both Firebird and SQLite which would both be worthy opponents.

一个后续：

我刚刚发现了 Synopse A大表 Bouchez ，其设计速度和几乎精确地符合我的问题的规格。我将在几个月内完成实施时首先尝试，并在此报告我的结果。

I just discovered Synopse Big Table by A. Bouchez which is designed for speed and meets the specs of my question almost precisely. I'll be trying it out first when I do my implementation in a few months and will report back here with my results.

稍后的追踪（2015年7月）

A much later followup (July 2015)

我从来没有尝试过Synopse Big Table。我已经坚持我的B *树到现在。但现在我升级到了Delphi XE8，并计划使用FireDAC和SQLite的数据库解决方案。

I never did get to try Synopse Big Table. I've stuck with my B* tree up to now. But now I've upgraded to Delphi XE8 and plan to go with the database solution using FireDAC with SQLite.

推荐答案

Synopse Big Table 作者： A。 Bouchez 。请参见他对我的其他人的回答问题关于SQLite / DISQLite。

Synopse Big Table by A. Bouchez. See his answer to my other question about SQLite/DISQLite.

当我第一次提出这个问题时，它甚至没有开发出来，但现在它是一个相当成熟和功能完备的单元。

It wasn't even developed when I first asked this question, but now it's a quite mature and fully functional unit.

这篇关于如何在Delphi中实现一个巨大但简单的索引StringList？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！