更改文件长度时，是否需要重新映射所有关联的MappedByteBuffers? [英] When changing the file length do I need to remap all associated MappedByteBuffers?

问题描述

我有一个小型且简单的存储系统，可以通过内存映射文件进行访问.因为我需要处理超过2GB的空间，所以我需要一个固定大小的MappedByteBuffer列表，例如2GB(出于各种原因，我使用的更少).然后，一切都相对简单了:一个缓冲区映射到某个空间，比如说1GB，当我需要更多时，我映射了一个新的MappedByteBuffer(文件会自动增加)，然后当我需要更多第三个缓冲区时，映射等等. /p>

但是随后，我读了 Java NIO书当我更改文件长度时可能会出现问题:

MappedByteBuffer直接反映与其关联的光盘文件.如果在映射生效时对文件进行了结构修改，则可能会导致奇怪的行为(确切的行为取决于操作系统和文件系统)MappedByteBuffer具有固定的大小，但映射到的文件是弹性的.特别是，如果在映射生效时文件大小发生变化，则某些或所有缓冲区可能变得不可访问，可能会返回未定义的数据，或者会引发未经检查的异常.当文件被内存映射时，请注意其他线程或外部进程如何处理文件.

我认为可能会出现问题，因为操作系统会在文件增加时移动文件，然后MappedByteBuffers可能指向无效空间(或者我误解了吗?)

因此，我没有在列表中添加新的MappedByteBuffer，而是在执行以下操作

1. 增加文件长度
2. 清除缓冲区列表(丢弃旧缓冲区，并希望这些缓冲区通过垃圾收集器释放.嗯，也许我应该通过cleaner.clean()显式清除所有缓冲区?)
3. 重新映射(用新缓冲区填充列表)

但是此过程的缺点是有时在映射时会失败

IOException: Operation not permitted
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:734)

为什么呢?因为清除缓冲区列表没有正确释放和清理缓冲区，并且不允许多个映射?我应该坚持旧的工作方法，而忽略书中的评论吗?

更新

• 在32位操作系统上拆分映射的优势在于，它可以更好地找到可用空间，并且出错的可能性较小( ref )
• 将映射分成较小的部分是一个优势，因为设置mmap可能会很昂贵( ref )
两种方法都不是干净的，而我的第二种方法应该可以工作，但是需要取消映射(将尝试使用普通的cleaner.clean hack强制释放).第一种方法应适用于系统(例如解决方案

根本原因是我的错:偶然地我重新映射了底层文件太多(仅通过小步骤就增加了容量).

但是，即使在这种极端情况下，当我重试失败的映射操作(+ System.gc + 5ms sleep->)时，我仍然能够最终修复IOException(不允许的操作)，这应该为jvm提供取消映射的机会缓冲区).现在我只看到大量重新映射，从而得出最终结论.

至少我对mmap有所了解:它非常依赖于OS +文件系统-也要感谢 auselen ！如果您想要一个干净的解决方案，则应按照他最初的建议，为每个文件使用一个MappedByteBuffer.但是，如果您需要大空间并且OS文件描述符限制太低，这也可能会引起问题.

最后但并非最不重要的一点是，我强烈建议您不要使用第一个解决方案，因为我找不到保证(仅在IBM OS中)，这在文件大小增加后不会使映射的缓冲区保持完整.

I've a small and simple storage system which is accessible through memory mapped files. As I need to address more than 2GB space I need a list of MappedByteBuffer with a fixed size like 2GB (I'm using less for different reasons). Then all is relativly simple: a buffer maps to a certain space say to 1GB and when I need more I map a new MappedByteBuffer (the file automatically increases), then when I need more a third buffer is mapped etc. This just worked.

But then I've read in the Java NIO book that when I change the file length problems could occur:

A MappedByteBuffer directly reflects the disc file with which it is associated. If the file is structurally modified while the mapping is in effect, strange behaviour can result (exact behaviour are OS and file system dependent) A MappedByteBuffer has a fixed size, but file it's mapped to is elastic. Specifically if a file's size changes while the mapping is in effect, some or all of the buffer may become inaccessible, undefined data could be returned, or unchecked exceptions could be thrown. Be careful about how files are manipulated by other threads or external processes when they are memory-mapped.

I think the problems could occur as the OS could move the file when it is increasing and the MappedByteBuffers then could point to an invalid space (or am I misinterpreting this?)

So, instead of adding a new MappedByteBuffer to the list I am now doing the following

1. increasing the file length
2. clearing the buffer list (throwing away the old buffers and hoping that the buffers where released via garbage collector. hmmh, probably I should clean all the buffers explicitly via cleaner.clean()?)
3. re-mapping (filling the list with fresh buffers)

BUT THIS procedure has the disadvantage of failing sometimes while mapping with

IOException: Operation not permitted
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:734)

Why that? Because clearing the buffer list did not properly release and clean the buffers and multiple mappings are not allowed? Should I just stick to the old working method and ignoring the comment in the book?

Update

• splitting the mapping on a 32bit OS has the advantage of better finding free space and less likely make an error (ref)
• splitting the mapping into smaller parts is an advantage as setting up the mmap could be costly (ref)
• both approaches are not clean while my second approach should work but will need an unmap (will try to force the release with the normal cleaner.clean hack). The first approach should work on systems (like ibm) where I could increase the file size but in general it won't work although I couldn't find the exact reason yet...
• cleanest way would be to use multiple files I fear (one file per MappedByteBuffer)

解决方案

The root cause was my fault: accidentally I remapped the underlying file too often (the capacity increased only via mini steps).

But even in this extreme situation I was able to finally fix the IOException (Operation not permitted) when I was retrying the failing mapping operation (+ System.gc + a 5ms sleep -> which should give the jvm the opportunity to unmap the buffer). Now I only saw massive numbers of remappings which leads to the final conclusion.

At least I learned a bit more about mmap: it is very OS+filesystem dependent - thanks also to auselen! If you like a clean solution you should use one MappedByteBuffer per file as initially suggested from him. But this could be also problematic if you need large space and your OS file descriptor limit is too low.

Last but not least I would strongly recommend against my first solution as I couldn't find a guarantee (only in the IBM OS ;)) which leaves the mapped buffer intact after a file size increase.

这篇关于更改文件长度时，是否需要重新映射所有关联的MappedByteBuffers?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！