Bitcask可以用于简单而高性能的文件存储吗? [英] Bitcask ok for simple and high performant file store?
问题描述
我正在寻找一种简单的方法来存储和检索数百万个xml文件.当前，所有操作都在文件系统中完成，这存在一些性能问题.
I am looking for a simple way to store and retrieve millions of xml files. Currently everything is done in a filesystem, which has some performance issues.
我们的要求是:
- 能够在批处理过程中存储数百万个xml文件. XML文件的大小上限为100 MB.
- 通过ID(例如文档URL)进行非常快速的随机查找
- Java和Perl均可访问
- 在最重要的Linux发行版和Windows上可用
我确实查看了几种NoSQL平台(例如CouchDB， Riak 等)，以及那些系统看起来很棒，它们几乎就像是蜜蜂般的过度杀伤力:
I did have a look at several NoSQL-Platforms (e.g. CouchDB, Riak and others), and while those systems look great, they seem almost like beeing overkill:
- 无需群集
- 不需要守护程序(服务")
- 不需要聪明的搜索功能
在深入研究Riak之后，我发现了Bitcask(请参见简介) ，这似乎正是我想要的.简介中介绍的基础知识确实很吸引人.但是不幸的是，没有办法通过java访问位桶仓库(或者在那里?)
Having delved deeper into Riak, I have found Bitcask (see intro), which seems like exactly what I want. The basics described in the intro are really intriguing. But unfortunately there is no means to access a bitcask repo via java (or is there?)
我的问题归结为
- 是以下正确的假设:Bitcask模型(仅追加写入，内存中的密钥管理)是存储/检索数百万个文档的正确方法
- 是否有Java可以替代Bitcask的可行替代方案? (我想起了BerkleyDB ...)
- (针对riak专家)与裸" Bitcask相比，Riak在开销，实施/管理/资源方面是否明智?
推荐答案
我认为Bitcask在您的用例中不能很好地工作.看起来Bitcask模型是为每个值的大小相对较小的用例而设计的.
I don't think that Bitcask is going to work well for your use-case. It looks like the Bitcask model is designed for use-cases where the size of each value is relatively small.
问题出在Bitcask的数据文件合并过程中.这涉及从多个旧数据文件"中复制所有实时值.到合并数据文件"中.如果您有数百万个值(每个值都在100Kb左右)，那么这就是疯狂的数据复制量.
The problem is in Bitcask's data file merging process. This involves copying all of the live values from a number of "older data file" into the "merged data file". If you've got millions of values in the region of 100Kb each, this is an insane amount of data copying.
请注意，以上内容假设XML文档的更新频率相对较高.如果很少进行更新和/或您可以应付大量的空间浪费"，那么合并可能只需要很少进行，或者根本不需要进行.
Note the above assumes that the XML documents are updated relatively frequently. If updates are rare and / or you can cope with a significant amount of space "waste", then merging may only need to be done rarely, or not at all.
这篇关于Bitcask可以用于简单而高性能的文件存储吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
