阵列等的磁盘存储 [英] disk storage of arrays etc
问题描述
有人在磁盘上存储数据方面有经验吗？我所拥有的是一个可以进行计算等的内存建模应用程序。基本上，数据存储为对象列表，这些对象具有嵌套的键值集合，例如Dictionary< int，字典< int，T >>。
Does anyone have experience with storing data on disk? What I have is an in-memory modelling application that can do calculations etc. Basically the data is stored as lists of objects, that have nested key-value collections like Dictionary< int, Dictionary< int, T>>.
现在，我使用SQL Server作为持久性层，但很少使用它的功能。因此，我想自己可以自己将数据写入/读取到磁盘上，以减少依赖性并简化安装。
Right now I use SQL-Server as a persistance layer but I use very few features of it. So I'm thinking I could write/ read the data to disk myself to reduce dependencies and ease installation.
因此，我编写了一个例程，将每个数组写入磁盘大致采用这种格式，其中文件中实际上没有单词 ObjId， Type， Valid和 Count，它们是byte []中的1、2、3、4整数，然后出现< ; int，T>对。 52来自4 * 4 + 3 *（4 + 8）。 （int为4个字节，double为8个字节）
So I wrote a little routine that writes each array to disk in roughly this format, where the words "ObjId", "Type", "Valid" and "Count" are not actually in the file, they're the 1st, 2nd, 3rd an 4th int in the byte[], then come < int, T > pairs. The 52 comes from 4 * 4 + 3 * (4 + 8). (4 bytes for int, 8 for double)
Bytes: 52
ObjId: 123
Valid: 234
Type: double
Count: 3
1 .23
2 .34
3 .45
在现实生活中，没有缩进等，它们都是很长的连续字节。
In real life there's no indentation etc, they're all sequential bytes in a long stream.
这很好，只写一次。但是，当我想在中间的某个地方写一个额外的值时，我必须重写整个内容。另外，我不能轻易地更新单个值。
This is fine, to write once. But when I want to write an extra value somewhere in the middle I have to rewrite the whole thing. Also I can't update a single value easily.
一种选择是将每个对象写入一个单独的文件中，因此我只需要重写它即可。但这似乎效率很低，因为我得到的文件大小为1kb，但磁盘上大小为4kB，所以我会浪费那里的空间。
One alternative is to write each object to a separate file so I would only have to rewrite that. But but that seems quite inefficient because I get files that are 1kb, but 4kB on disk so I'd be wasting space there.
那么，我需要做些什么，才能递增地将此文件写入磁盘？我知道SqlServer在写入数据的地方有页面，这是要走的路吗？
So what do I need to to do, to be able to incrementenally write to this file on disk? I know SqlServer has 'pages' where it writes data, is that the way to go?
有没有准备好解决此类问题的库？也许有些虚拟文件可以让我将它们视为单独的byte []，但将存储作为单个psysical文件处理？理想情况下是压缩的。（将其推入，但谁知道。。我之前很惊讶：-）
Is there any library ready to go for this type of problem? Maybe some virtual file that will let me treat them as seperate byte[] but handles the storage as a single psysical file? Ideally compressed.. (pushing it, but who knows.. I've been surprised before :-)
预先感谢，
Gert-Jan
推荐答案
如果您不需要RDBMS的开销，则可以使用键值数据库（例如Berkeley DB）。此处有一个C＃接口：
If you don't want the overhead of an RDBMS, you could use a key-value database like Berkeley DB. There is a C# interface for it here:
每个数组可以有一个条目，并在需要时重写它。数据库文件的其余部分将保持不变，因此它比重写整个文件要快得多。
You can have one entry for each array, and just rewrite that when you need to. The rest of the database file will be unchanged so it's much faster than rewriting the whole file.
您可以重复使用已经实现的序列化逻辑。数组。您需要添加的是每个数组的唯一键。
You can reuse the serialization logic you've already implemented when you write out an array. All you need to add is a unique key for each array.
这篇关于阵列等的磁盘存储的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
