Java MemoryMapping大文件 [英] Java MemoryMapping big files

问题描述

MappedByteBuffer到2GIG的Java限制使得用于映射大文件变得非常棘手。通常推荐的方法是使用一个MappedByteBuffer数组并将其索引：

  long PAGE_SIZE = Integer.MAX_VALUE; 
 MappedByteBuffer [] buffers; 
 $ b $ private int getPage（long offset）{
 return（int）（offset / PAGE_SIZE）
} 
 $ b private int getIndex（long offset）{ 
 return（int）（offset％PAGE_SIZE）; 
 
 $ b $ public byte get（long offset）{
 return buffers [getPage（offset）]。get（getIndex（offset））; 
 
 
 
 
 
 $ b 
这可以是一个单字节的工作，但需要重写很多代码如果你想处理更大的读/写，并且需要跨越边界（getLong（）或者get（byte []））。
 
 
问题：什么是最好的实践这种情况下，你知道任何工作的解决方案/代码，可以重新使用，而不需要重新发明轮子？
 
解决方案
您是否检出了 dsiutil的 ByteBufferInputStream  ？
 
 
 
 Javadoc 
 
 
 
这个主要用处类是创建真正基于 MappedByteBuffer 的输入流的可能。
 
 
 
特别的，工厂方法 map（FileChannel，FileChannel.MapMode）将把整个文件存储到一个 ByteBuffer 的数组中将数组暴露为ByteBufferInputStream。这使得访问容易映射的大于2GiB的文件成为可能。  
 
 
 
 
  long length（）  long position（） 
 
 
  void position（long newPosition） 
 
 
 
 
 
这是你在想什么吗？  LGPL也是。
 
 
 
 
 
The Java limitation of MappedByteBuffer to 2GIG make it tricky to use for mapping big files. The usual recommended approach is to use an array of MappedByteBuffer and index it through:
long PAGE_SIZE = Integer.MAX_VALUE;
MappedByteBuffer[] buffers;

private int getPage(long offset) {
    return (int) (offset / PAGE_SIZE)
}

private int getIndex(long offset) {
    return (int) (offset % PAGE_SIZE);
}

public byte get(long offset) {
    return buffers[getPage(offset)].get(getIndex(offset));
}
this can be a working for single bytes, but requires rewriting a lot of code if you want to handle read/writes that are bigger and require crossing boundaries (getLong() or get(byte[])).

The question: what is your best practice for these kind of scenarios, do you know any working solution/code that can be re-used without re-inventing the wheel?
解决方案
Have you checked out dsiutil's ByteBufferInputStream?

Javadoc

The main usefulness of this class is that of making it possible creating input streams that are really based on a MappedByteBuffer.

In particular, the factory method map(FileChannel, FileChannel.MapMode) will memory-map an entire file into an array of ByteBuffer and expose the array as a ByteBufferInputStream. This makes it possible to access easily mapped files larger than 2GiB. 


long length()
long position()
void position(long newPosition)


Is that something you were thinking of?  It's LGPL too.



                        
这篇关于Java MemoryMapping大文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！