用于DB实现的内存映射的BappedBuffer或直接ByteBuffer？ [英] Memory-Mapped MappedByteBuffer or Direct ByteBuffer for DB Implementation?
问题描述
由于所有的上下文，这看起来像一个漫长的问题。下面的小说里有两个问题。感谢您抽出时间阅读并提供帮助。
情况
我正在开发一个可扩展的数据存储实现，可以支持在32位或64位系统上处理从几KB到TB或更多数据文件的数据文件。
数据存储区使用写时复制设计;始终将新的或修改的数据附加到数据文件的末尾，而不要对现有数据进行就地编辑。
系统可以托管1个或更多数据库;每个由磁盘上的文件表示。
实现的细节并不重要;唯一重要的细节是，我需要不断地追加到文件，并将其从KB扩展到MB到GB到TB，同时随机跳过文件进行读取操作以回答客户端请求。
First-Thoughts
乍一看，我知道我想使用内存映射文件，所以我可能会将数据的内存状态有效地管理到主机操作系统上，而不需要我的代码。
然后，我所有的代码都需要担心的是在写入时序列化附加到文件的操作，并允许任意数量的同时读取器在文件中寻找答案请求。
设计
因为单个数据文件可以超出MappedByteBuffer的2GB限制，我希望我的设计必须包含一个抽象层，它需要一个写入偏移并将其转换成特定2GB段的偏移量。
到目前为止这么好...
问题
这是我在哪里开始挂断，认为用不同的设计（如下所述）可能是更好的方法。
从阅读到20左右内存映射这里的相关问题在这里，似乎mmap调用是敏感的，希望在分配时连续运行内存。所以，例如，在32位主机操作系统上，如果我试图mmap一个2GB的文件，由于内存碎片，我的机会很小，映射将成功，而我应该使用像一系列的128MB映射来拉一个整体文件中。
当我想到这个设计，甚至说使用1024MB的mmap大小，一个DBMS托管了几个由1TB文件表示的巨大数据库，我现在在内存中具有数千个内存映射区域，而在我自己的Windows 7测试中，尝试在多GB文件中创建几百mmaps，我没有遇到异常，我实际上得到每当我试图分配太多的JVM时，JVM会出现故障，在某种情况下，我的Windows 7机器中的视频可以使用我以前从未见过的操作系统错误弹出窗口来重新初始化。
无论你永远不会处理大文件或这是一个有创意的例子的论点，我可以用这些类型的代码来编写一些东西效果使我的内部警报处于高度警戒状态，并考虑到另一种含义（下文）。
发现这个问题的BESIDES，我对内存映射文件的理解是，我必须每次文件生成时重新创建映射，所以在这个文件的情况下，只是在设计中附加，它的字面上不断增长。
我可以打架这在某种程度上通过以块的形式增长（每次8MB），并且每8MB重新创建一次映射，但是需要不断地重新创建这些映射，所以我特别紧张，特别是没有明确的 Java中支持的映射功能。
问题＃1/2
鉴于我的所有发现，至于这一点，我会将内存映射文件视为主要的一个很好的解决方案只读解决方案或只读解决方案，但不需要重写解决方案，因为需要不断重新创建映射。
但是，我环顾四周在我周围的解决方案，如MongoDB包含内存映射文件在整个地方，我觉得我在这里缺少一些核心组件（我知道它分配在一个像2GB的范围，一次，所以我想象他们正在围绕re这个逻辑的映射成本和帮助维护顺序运行在磁盘上）。
在这一点上，我不知道问题是Java是否缺少一个unmap操作这使得这样做更加危险，不适合我的使用，或者如果我的理解不正确，有人可以指向我。
替代设计 / p>
对于上面提出的内存映射的另一种设计，如果我对mmap的理解是正确的，我将会介绍如下：
定义一个合理的可配置大小（2,4,8,16,32,64,128KB大致）的直接ByteBuffer ，使其与任何主机平台轻松兼容（不需要担心DBMS本身造成颠簸场景）并使用原始FileChannel，执行特定偏移量读取文件1个buffer-capacity-chunk，一次完全删除内存映射文件。
缺点是现在我的代码不得不担心像我从文件中读取足够的内容以加载完整的记录。
另一个缺点是，我不会使用操作系统的虚拟内存逻辑，让它自动为我保留更多的热数据在内存中;相反，我只是希望操作系统使用的文件缓存逻辑足够大，可以为我做一些有用的事情。
问题＃2/2
我希望能够确认我对这一切的理解。例如，也许文件缓存太棒了，在这两种情况下（内存映射或直接读取），主机操作系统将保持与我的热点数据一样多或许我对内存映射文件（连续内存）的敏感要求的理解是不正确的，我可以忽略所有的文件。
您可能对 https://github.com/peter-lawrey/Java-Chronicle
在此我创建多个内存映射到同一个文件尺寸是2到1 GB的权力）文件可以是任何大小（达到硬盘大小）
它还创建一个索引，所以你可以随机找到任何记录，每个记录可以是任何大小。
它可以在进程之间共享，并用于进程之间的低延迟事件。
如果要使用大量数据，我假设您正在使用64位操作系统。在这种情况下，MappedByteBuffer的列表将是您所需要的。使用合适的工具来完成工作是有道理的。 ;）
我发现它的性能很好，即使数据大小在你的主内存大小的10倍左右（我正在使用一个快速的SSD驱动器，所以YMMV）
This looks like a long question because of all the context. There are 2 questions inside the novel below. Thank you for taking the time to read this and provide assistance.
Situation
I am working on a scalable datastore implementation that can support working with data files from a few KB to a TB or more in size on a 32-bit or 64-bit system.
The datastore utilizes a Copy-on-Write design; always appending new or modified data to the end of the data file and never doing in-place edits to existing data.
The system can host 1 or more database; each represented by a file on-disk.
The details of the implementation are not important; the only important detail being that I need to constantly append to the file and grow it from KB, to MB, to GB to TB while at the same time randomly skipping around the file for read operations to answer client requests.
First-Thoughts
At first glance I knew I wanted to use memory-mapped files so I could push the burden of efficiently managing the in-memory state of the data onto the host OS and out of my code.
Then all my code needs to worry about is serializing the append-to-file operations on-write, and allowing any number of simultaneous readers to seek in the file to answer requests.
Design
Because the individual data-files can grow beyond the 2GB limit of a MappedByteBuffer, I expect that my design will have to include an abstraction layer that takes a write offset and converts it into an offset inside of a specific 2GB segment.
So far so good...
Problems
This is where I started to get hung up and think that going with a different design (proposed below) might be the better way to do this.
From reading through 20 or so "memory mapped" related questions here on SO, it seems mmap calls are sensitive to wanting contiguous runs of memory when allocated. So, for example, on a 32-bit host OS if I tried to mmap a 2GB file, due to memory fragmentation, my chances are slim that mapping will succeed and instead I should use something like a series of 128MB mappings to pull an entire file in.
When I think of that design, even say using 1024MB mmap sizes, for a DBMS hosting up a few huge databases all represented by say 1TB files, I now have thousands of memory-mapped regions in memory and in my own testing on Windows 7 trying to create a few hundred mmaps across a multi-GB file, I didn't just run into exceptions, I actually got the JVM to segfault every time I tried to allocate too much and in one case got the video in my Windows 7 machine to cut out and re-initialize with a OS-error-popup I've never seen before.
Regardless of the argument of "you'll never likely handle files that large" or "this is a contrived example", the fact that I could code something up like that with those type of side effects put my internal alarm on high-alert and made consider an alternative impl (below).
BESIDES that issue, my understanding of memory-mapped files is that I have to re-create the mapping every time the file is grown, so in the case of this file that is append-only in design, it literally constantly growing.
I can combat this to some extent by growing the file in chunks (say 8MB at a time) and only re-create the mapping every 8MB, but the need to constantly be re-creating these mappings has me nervous especially with no explicit unmap feature supported in Java.
Question #1 of 2
Given all of my findings up to this point, I would dismiss memory-mapped files as a good solution for primarily read-heavy solutions or read-only solutions, but not write-heavy solutions given the need to re-create the mapping constantly.
But then I look around at the landscape around me with solutions like MongoDB embracing memory-mapped files all over the place and I feel like I a missing some core component here (I do know it allocs in something like 2GB extents at a time, so I imagine they are working around the re-map cost with this logic AND helping to maintain sequential runs on-disk).
At this point I don't know if the problem is Java's lack of an unmap operation that makes this so much more dangerous and unsuitable for my uses or if my understanding is incorrect and someone can point me North.
Alternative Design
An alternative design to the memory-mapped one proposed above that I will go with if my understanding of mmap is correct is as follows:
Define a direct ByteBuffer of a reasonable configurable size (2, 4, 8, 16, 32, 64, 128KB roughly) making it easily compatible with any host platform (don't need to worry about the DBMS itself causing thrashing scenarios) and using the original FileChannel, perform specific-offset reads of the file 1 buffer-capacity-chunk at a time, completely forgoing memory-mapped files at all.
The downside being that now my code has to worry about things like "did I read enough from the file to load the complete record?"
Another down-side is that I don't get to make use of the OS's virtual memory logic, letting it keep more "hot" data in-memory for me automatically; instead I just have to hope the file cache logic employed by the OS is big enough to do something helpful for me here.
Question #2 of 2
I was hoping to get a confirmation of my understanding of all of this.
For example, maybe the file cache is fantastic, that in both cases (memory mapped or direct reads), the host OS will keep as much of my hot data available as possible and the performance difference for large files is negligible.
Or maybe my understanding of the sensitive requirements for memory-mapped files (contiguous memory) are incorrect and I can ignore all that.
You might be interested in https://github.com/peter-lawrey/Java-Chronicle
In this I create multiple memory mappings to the same file (the size is a power of 2 up to 1 GB) The file can be any size (up to the size of your hard drive)
It also creates an index so you can find any record at random and each record can be any size.
It can be shared between processes and used for low latency events between processes.
I make the assumption you are using a 64-bit OS if you want to use large amounts of data. In this case a List of MappedByteBuffer will be all you ever need. It makes sense to use the right tools for the job. ;)
I have found it performance well even with data sizes around 10x your main memory size (I was using a fast SSD drive so YMMV)
这篇关于用于DB实现的内存映射的BappedBuffer或直接ByteBuffer？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
