FILESYSTEM vs SQLITE，同时存储多达10M的文件 [英] FILESYSTEM vs SQLITE, while storing up-to 10M files

问题描述

我想存储多达10M的文件，2TB存储单元。我需要的唯一属性限制为文件名及其内容（数据）。

I would like to store up-to 10M files, 2TB storage unit. The only properties which I need restricted to filenames, and their contents (data).

文件最大长度为100MB，大部分小于1MB。需要删除文件的能力，并且写入和读取速度应该是一个优先 - 而低的存储效率，恢复或完整性方法，不需要。

The files max-length is 100MB, most of them are less than 1MB. The ability of removing files is required, and both writing and reading speeds should be a priority - while low storage efficiency, recovery or integrity methods, are not needed.

关于NTFS，但它的大部分功能是不需要的，而不能被禁用，并被认为是一个开销关注，其中几个是：创建日期，修改日期，条目，日志和当然的权限。

I thought about NTFS, but most of its features are not needed, while can't be disabled and considered to be an overhead concern, a few of them are: creation date, modification date, attribs, journal and of course permissions.

由于不需要文件系统的本地特性，你建议我使用SQLITE这个要求吗？还是有一个明显的缺点，我应该注意？ （可以猜到删除文件将是一项复杂的任务？）

Due to the native features of a filesystem which are not needed, would you suggest I'll use SQLITE for this requirement? or there's an obvious disadvantage that I should be aware about? (one would guess that removing files will be a complicated task?)

（SQLITE将通过C api）

(SQLITE will be via the C api)

我的目标是使用更适合的解决方案来获得性能。提前感谢 - Doori Bar

My goal is to use a more suited solution to gain performance. Thanks in advance - Doori Bar

推荐答案

如果您的主要要求是性能，请使用本机文件系统。 DBMS不太适合处理大型BLOB，所以SQLite不是一个选择你根本不知道（甚至不知道为什么每个人都认为SQLite是一个插件，每个洞）。

If your main requirement is performance, go with native file system. DBMS are not well suited for handling large BLOBs, so SQLite is not an option for you at all (don't even know why everybody considers SQLite to be a plug for every hole).

要提高NTFS（或您选择的任何其他文件系统）的性能，请勿将所有文件放入单个文件夹，而是将文件按文件名的前N个字符分组，或也通过扩展。

To improve performance of NTFS (or any other file system you choose) don't put all files into single folder, but group files by first N characters of their file names, or also by extension.

市场上还有一些其他文件系统，也许其中一些提供了禁用一些使用的功能的可能性。您可以检查维基百科上的比较，并检查它们。

Also there exist some other file systems on the market and maybe some of them offer possibility to disable some of used features. You can check the comparison on Wikipedia and check them.

更正：我做了一些测试（不是非常广泛的），显示没有性能的好处分组文件到大多数类型的操作的子目录，NTFS相当有效地处理26 ^ 4空文件命名从AAAA到ZZZZ在单个目录。所以你需要检查你的特定文件系统的效率。

Correction: I've made some tests (not very extensive though) that show no performance benefit in grouping files into subdirectories for most types of operations, and NTFS quite efficiently handled 26^4 empty files named from AAAA to ZZZZ in a single directory. So you need to check efficiency for your particular file system.

这篇关于FILESYSTEM vs SQLITE，同时存储多达10M的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！