android mmap 因内存不足而失败 [英] android mmap fails with out of memory
问题描述
我到处寻找答案，但我认为我已经达到了我能找到的极限.我的问题似乎与此有关:Android NDK mmap 调用在升级到 Lollipop 后在 32 位设备上中断，但未提供任何答案.
I've searched everywhere for an answer but I think I'm hitting the limits of what I can find. My question seems somewhat related to this one : Android NDK mmap call broken on 32-bit devices after upgrading to Lollipop but no answer has been provided.
我的问题是我尝试通过 mmap 调用从文件中映射 457232384 个字节.在装有 Android 5.1.1 的两台不同设备(三星 Galaxy Note 3 和 OnePlus One，每台 3GB RAM)上，调用失败并显示 errno 12内存不足".实际上，当我尝试分配超过 300MB 的内存时，调用会失败.313524224 字节 (299MB) 有效，314572800 (300MB) 无效.
My problem is that I try to memory map 457232384 bytes from a file through a mmap call. On two different devices (Samsung Galaxy Note 3 & OnePlus One, 3GB RAM each) with Android 5.1.1, that call fails with errno 12 "Out of memory". Actually, the call fails when I try to allocate more than 300MB of memory. 313524224 bytes (299MB) works, 314572800 (300MB) won't.
问题是，同样的调用可以在第三台设备上运行，该设备仍然运行 Android 4.4.2.更奇怪的是，这个调用适用于带有 SDK 21 (Android 5.0) 的 Android ARM 模拟器.不用说，可以毫无问题地加载相同数量的数据(未映射).
Thing is, the very same call works on a third device which stayed on Android 4.4.2. Even stranger, this call works on the Android ARM emulator with SDK 21 (Android 5.0). Needless to say, the same amount of data (not mmap'ed) can be loaded without any issue.
dmesg 向我报告:
dmesg reports this to me:
<3>[ 1137.488411] [0:Thread-298: 4267] arch_get_unmapped_area (TASK_SIZE - len < addr) len=457232384 task size=3204448256 pid=4267 do_align=0 addr=3034054656 mmap_base=3069939712
尝试映射文件的函数(来自 openfst)如下:
The function (from openfst) which tries to map the file is the following:
MappedFile* MappedFile::Map(istream* s, const FstReadOptions &opts,
size_t size) {
size_t pos = s->tellg();
if (opts.mode == FstReadOptions::MAP && pos >= 0 &&
pos % kArchAlignment == 0) {
int fd = open(opts.source.c_str(), O_RDONLY);
if (fd != -1) {
int pagesize = getpagesize();
off_t offset = pos % pagesize;
off_t upsize = size + offset;
void *map = mmap(NULL, upsize, PROT_READ, MAP_SHARED, fd, pos - offset);
char *data = reinterpret_cast<char*>(map);
if (close(fd) == 0 && map != MAP_FAILED) {
MemoryRegion region;
region.mmap = map;
region.size = upsize;
region.data = reinterpret_cast<void*>(data + offset);
MappedFile *mmf = new MappedFile(region);
s->seekg(pos + size, ios::beg);
if (s) {
VLOG(1) << "mmap'ed region of " << size << " at offset " << pos
<< " from " << opts.source.c_str() << " to addr " << map;
return mmf;
}
delete mmf;
} else {
LOG(INFO) << "Mapping of file failed: " << strerror(errno);
}
}
}
// If all else fails resort to reading from file into allocated buffer.
if (opts.mode != FstReadOptions::READ) {
LOG(WARNING) << "File mapping at offset " << pos << " of file "
<< opts.source << " could not be honored, reading instead.";
}
MappedFile* mf = Allocate(size);
if (!s->read(reinterpret_cast<char*>(mf->mutable_data()), size)) {
delete mf;
return NULL;
}
return mf;
}
每次从 mmap 返回都是 MAP_FAILED.
Return from mmap is MAP_FAILED everytime.
有人对我在哪里可以解决我的问题有建议吗?谢谢！
Does someone has suggestions on where can I look to solve my issue? Thanks!
这是/proc/self/maps 在臭名昭著的 mmap 调用之后的内容:http://pastebin.com/1864jZC2
here is the content of /proc/self/maps right after the infamous mmap call : http://pastebin.com/1864jZC2
一点差距分析:
Gap between 00000000 and 12c00000 (diff = 314572800 bytes, 300 MB)
Gap between 42c00000 and 55281000 (diff = 308809728 bytes, 294.50390625 MB)
Gap between 67e80000 and 67ea4000 (diff = 147456 bytes, 0.140625 MB)
Gap between 7778b000 and 77800000 (diff = 479232 bytes, 0.45703125 MB)
Gap between 77a80000 and 77a82000 (diff = 8192 bytes, 0.0078125 MB)
Gap between 77c00000 and 77c04000 (diff = 16384 bytes, 0.015625 MB)
Gap between 78080000 and 780b7000 (diff = 225280 bytes, 0.21484375 MB)
Gap between 79ac1000 and 79ac2000 (diff = 4096 bytes, 0.00390625 MB)
Gap between 7db70000 and 7db71000 (diff = 4096 bytes, 0.00390625 MB)
Gap between 7e000000 and 7e001000 (diff = 4096 bytes, 0.00390625 MB)
Gap between 7e0fe000 and 7e0ff000 (diff = 4096 bytes, 0.00390625 MB)
Gap between 7e145000 and 7e146000 (diff = 4096 bytes, 0.00390625 MB)
Gap between b6fb9000 and be6ff000 (diff = 125067264 bytes, 119.2734375 MB)
Gap between beeff000 and ffff0000 (diff = 1091506176 bytes, 1040.94140625 MB)
在@fadden 的回答中对我有用的解决方案.
the solution that worked for me in the comments of @fadden's answer.
TL;DR:将 dalvik.vm.heapsize 设置为 512m.
TL;DR: set dalvik.vm.heapsize to 512m.
推荐答案
mmap()失败后转储一份/proc/self/maps的副本(只要打开从您的代码中提取文件并将内容复制到临时文件中).由于 ASLR，您可能无法获得较大的连续虚拟地址范围.
Dump a copy of /proc/self/maps after mmap() fails (just open the file from your code and copy the contents to a temp file). You may be having a problem getting a large contiguous virtual address range due to ASLR.
Android 中的一些 Zip 文件处理代码使用 mmap() 来映射整个文件，并在内存中对其进行操作.有一天，有人创建了一个 1GB 的 Zip 文件，但无法打开它.虽然进程的虚拟地址空间有足够的空闲页面，但没有足够的连续页面来创建单个线性映射.(IIRC，解决方案是 mmap() 只是中央目录.)
Some of the Zip file handling code in Android was using mmap() to map the entire file, and just operate on it in memory. One day somebody created a 1GB Zip file and was unable to open it. While the process' virtual address space had enough free pages, there weren't enough contiguous pages to create a single linear mapping. (IIRC, the solution was to mmap() just the central directory.)
maps 输出将显示您的进程的地址空间是什么样的.300MB 对于 ASLR/碎片来说似乎有点低，但这是一个开始寻找的好地方，并且可以解释不一致的行为.
The maps output will show you what your process' address space looks like. 300MB seems a bit low for ASLR / fragmentation to be an issue, but it's a good place to start looking, and might explain the inconsistent behavior.
这篇关于android mmap 因内存不足而失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
