内存映射缓冲区的增量分配和失败(1 GB大小) [英] Incremental allocation of memory mapped buffer and failure at 1 GB size
问题描述
我创建了以下演示以查看MMF行为(我想将其用作长值的超大型数组).
I have created the following demo to see the MMF begaviour (I want to use it as a very large array of long values).
import java.nio._, java.io._, java.nio.channels.FileChannel
object Index extends App {
val formatter = java.text.NumberFormat.getIntegerInstance
def format(l: Number) = formatter.format(l)
val raf = new RandomAccessFile("""C:\Users\...\Temp\96837624\mmf""", "rw")
raf.setLength(20)
def newBuf(capacity: Int) = {
var bytes= 8.toLong*capacity
println("new buf " + format(capacity) + " words = " + format(bytes) + " bytes")
// java.io.IOException: "Map failed" at the following line
raf.getChannel.map(FileChannel.MapMode.READ_WRITE, 0, bytes).asLongBuffer()
}
(1 to 100 * 1000 * 1000).foldLeft(newBuf(2): LongBuffer){ case(buf, i) =>
if (Math.random < 0.000009) println(format(buf.get(buf.position()/2)))
(if (buf.position == buf.capacity) {
val p = buf.position
val b = newBuf(buf.capacity * 2)
b.position(p) ; b
} else buf).put(i)
}
raf.close
输出失败
16,692,145
16,741,940
new buf 67,108,864
[error] (run-main-1) java.io.IOException: Map failed
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:907)
我看到已创建一个512 MB的文件，系统似乎无法将其扩展到1 GB.
I see a 512-MB file created and system seems failed to expand it to 1 GB.
但是，如果我不是使用2个长字的初始大小foldLeft(newBuf(2))，而是使用64M个长字newBuf(64*1024*1027)，则运行时会成功创建1GB文件，而在尝试使用2p创建长2GB文件时会失败
If, however, instead of initial size of 2 long words, foldLeft(newBuf(2)), I use 64M long words, newBuf(64*1024*1027), runtime succeeds creating 1GB file and fails when it tries to create 2GB file with
new buf 268 435 458 words = 2 147 483 664 bytes
java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
at sun.nio.ch.FileChannelImpl.map(Unknown Source)
我用64位jvm运行它.
I ran it with 64-bit jvm.
我也不确定如何关闭缓冲区以将其释放以供以后在sbt中使用，并确保数据最终会出现在文件中.该机制看起来完全不可靠.
I am also not sure how to close the buffer to release it for later application in sbt and be sure that data will ultimately appear in the file. The mechanism looks utterly unreliable.
推荐答案
好吧，有一天的实验表明，无论如何，32位JVM都会因IOException: Map failed出现1 GB的故障而失败.为了通过在64位计算机上进行映射来避开Size exceeds Integer.MAX_VALUE，应该使用负担得起的大小的多个缓冲区，例如每个100 mb都可以.这是因为缓冲区是通过整数寻址的.
Ok, one day of experiments has demonstrated that 32-bit JVM fails with IOException: Map failed at 1 GB no matter what. In order to circumvent the Size exceeds Integer.MAX_VALUE with mapping at 64-bit machines, one should should use multiple buffers of affordable size, e.g. 100 mb each is fine. That is because buffers are addressed by integer.
对于这个问题有什么看法，您可以同时在内存中打开所有此类缓冲区，即，无需分配一个缓冲区=> null，然后分配下一个有效增加文件大小，如以下演示所示
What regards this question, you can keep all such buffers open in memory simultaneously, i.e. there is no need to close one buffer => null before you allocate the next effectively increment the file size, as the following demo demonstrates
import Utils._, java.nio._, java.io._, java.nio.channels.FileChannel
object MmfDemo extends App {
val bufAddrWidth = 25 /*in bits*/ // Every element of the buff addresses a long
val BUF_SIZE_WORDS = 1 << bufAddrWidth ; val BUF_SIZE_BYTES = BUF_SIZE_WORDS << 3
val bufBitMask = BUF_SIZE_WORDS - 1
var buffers = Vector[LongBuffer]()
var capacity = 0 ; var pos = 0
def select(pos: Int) = {
val bufn = pos >> bufAddrWidth // higher bits of address denote the buffer number
//println(s"accessing $pos = " + (pos - buf * wordsPerBuf) + " in " + buf)
while (buffers.length <= bufn) expand
pass(buffers(bufn)){_.position(pos & bufBitMask)}
}
def get(address: Int = pos) = {
pos = address +1
select(address).get
}
def put(value: Long) {
//println("writing " + value + " to " + pos)
select(pos).put(value) ; pos += 1
}
def expand = {
val fromByte = buffers.length.toLong * BUF_SIZE_BYTES
println("adding " + buffers.length + "th buffer, total size expected " + format(fromByte + BUF_SIZE_BYTES) + " bytes")
// 32bit JVM: java.io.IOException: "Map failed" at the following line if buf size requested is larger than 512 mb
// 64bit JVM: IllegalArgumentException: Size exceeds Integer.MAX_VALUE
buffers :+= fc.map(FileChannel.MapMode.READ_WRITE, fromByte, BUF_SIZE_BYTES).asLongBuffer()
capacity += BUF_SIZE_WORDS
}
def rdAll(get: Int => Long) {
var firstMismatch = -1
val failures = (0 until parse(args(1))).foldLeft(0) { case(failures, i) =>
val got = get(i)
if (got != i && firstMismatch == -1) {firstMismatch = i; println("first mismatch at " +format(i) + ", value = " + format(got))}
failures + ?(got != i, 1, 0)
} ; println(format(failures) + " mismatches")
}
val raf = new RandomAccessFile("""C:\Temp\mmf""", "rw")
val fc = raf.getChannel
try {
if (args.length < 1) {
println ("usage1: buf_gen <len in long words>")
println ("usage1: raf_gen <len in long words>")
println("example: buf_gen 30m")
println("usage2: raf_rd <size in words>")
println("usage3: buf_rd <size in words>")
} else {
val t1 = System.currentTimeMillis
args(0) match {
case "buf_gen" => raf.setLength(0)
(0 until parse(args(1))) foreach {i => put(i.toLong)}
case "raf_gen" => raf.setLength(0)
(0 until parse(args(1))) foreach {i =>raf.writeLong(i.toLong)}
//fc.force(true)
case "rd_raf" => rdAll{i => raf.seek(i.toLong * 8) ; raf.readLong()}
case "rd_buf" => rdAll(get)
case u =>println("unknown command " + u)
} ; println("finished in " + (System.currentTimeMillis - t1) + " ms")
}
} finally {
raf.close ; fc.close
buffers = null ; System.gc /*GC needs to close the buffer*/}
}
object Utils {
val formatter = java.text.NumberFormat.getIntegerInstance
def format(l: Number) = formatter.format(l)
def ?[T](sel: Boolean, a: => T, b: => T) = if (sel) a else b
def parse(s: String) = {
val lc = s.toLowerCase()
lc.filter(_.isDigit).toInt *
?(lc.contains("k"), 1000, 1) *
?(lc.contains("m"), 1000*1000, 1)
}
def eqa[T](a: T, b: T) = assert(a == b, s"$a != $b")
def pass[T](a: T)(code: T => Unit) = {code(a) ; a}
}
至少在Windows中.使用此程序，我设法创建了比我的机器内存更大的mmf文件(更不用说JVM的-Xmx了，它们在这些问题上根本不起作用).只需用鼠标在Windows控制台中选择一些文本来减慢文件生成速度(程序会暂停直到您释放选择内容)，否则Windows会将所有其他性能至关重要的人员逐出页面文件，并且您的PC会死机.
at least in Windows. Using this program, I have managed to create mmf file larger than my machine memory (not to speak about JVM's -Xmx, which play no role at all in these matters). Just slow down the file generation selecting some text in the Windows console with mouse (program will pause until you release the selection) because otherwise Windows will evict all other performance critical staff to page file and your PC will die in thrashing.
顺便说一句，尽管我只写文件末尾，但PC仍死机，Windows可能会驱逐未使用的千兆字节块.另外，我注意到我正在写的块实际上已被读取
BTW, PC dies in thrashing despite I write only into the end of file and Windows could evict my unused gigabyte blocks. Also, I have noticed that the block I am writing is actually read
以下输出
adding 38th buffer, total size expected 12,480,000,000 bytes
adding 39th buffer, total size expected 12,800,000,000 bytes
伴随以下系统请求
5:24,java,"QueryStandardInformationFile",mmf,"SUCCESS","AllocationSize: 12 480 000 000, EndOfFile: 12 480 000 000, NumberOfLinks: 1, DeletePending: False, Directory: False"
5:24,java,"SetEndOfFileInformationFile",mmf,"SUCCESS","EndOfFile: 12 800 000 000"
5:24,java,"SetAllocationInformationFile",mmf,"SUCCESS","AllocationSize: 12 800 000 000"
5:24,java,"CreateFileMapping",mmf,"FILE LOCKED WITH WRITERS","SyncType: SyncTypeCreateSection, PageProtection: "
5:24,java,"QueryStandardInformationFile",mmf,"SUCCESS","AllocationSize: 12 800 000 000, EndOfFile: 12 800 000 000, NumberOfLinks: 1, DeletePending: False, Directory: False"
5:24,java,"CreateFileMapping",mmf,"SUCCESS","SyncType: SyncTypeOther"
5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 000 000, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 032 768, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 065 536, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 098 304, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 131 072, Length: 20 480, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
skipped 9000 reads
5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 836 160, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 868 928, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 901 696, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 934 464, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 967 232, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
但这是另一个故事.
事实证明，该答案与彼得·劳瑞(Peter Lawrey)的重复相同，只是我的问题专门针对地图失败"而映射大缓冲区时，超出了整数范围"，而最初的问题与JVM中的OutOfMem有关，这与I/O无关.
It turns out that this answer is duplicate of Peter Lawrey's except that mine question is dedicated to 'Map failed' and 'Integer range exceeded' when mapping large buffers whereas original question is concerned with OutOfMem in JVM, which has nothing to do with I/O.
这篇关于内存映射缓冲区的增量分配和失败(1 GB大小)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
