将几个大文件映射到内存中 [英] Mapping of several big files into memory

问题描述

在我们的应用程序中，我们必须能够将多个(即最多 4 个)文件映射到内存中(通过 mapViewOfFile).很长一段时间以来，这都不是问题，但是随着过去几年文件变得越来越大，现在内存碎片阻止我们映射这些大文件(文件将约为 200 MB).如果当时没有加载其他文件，则问题可能已经存在.

In our application we have to be able to map several (i.e. maybe up to 4) files into memory (via mapViewOfFile). For a long time this has not been a problem, but as the files were getting bigger and bigger over the last years, now memory fragmentation prevents us from mapping those big files (files will be about 200 MB). The problem may already exist if no other files are loaded at that moment.

我现在正在寻找确保映射始终成功的方法.因此，我想在程序启动时为映射保留一块内存，从而减少碎片化的影响.

I am now looking for a way t make sure that the mapping is always successful. Therefor I wanted to reserve a block of memory at program start only for the mapping and that would therefor suffer much less from the fragmentation.

我的第一种方法是HeapCreate一个私有堆，然后我会HeapAlloc一块足够大的内存来保存一个文件的映射，然后使用MapViewOfFileEx 带有该块的地址.因为地址必须与内存分配粒度相匹配.但映射仍然失败，错误代码为 ERROR_INVALID_ADDRESS (487).

My first approach was to HeapCreate a private heap, I would then HeapAlloc a block of memory large enough to hold the mapping for one file and then use MapViewOfFileEx with the address of that block. Of cause the address would have to match the memory allocation granularity. But the mapping still failed with error code ERROR_INVALID_ADDRESS (487).

接下来我用 VirtualAloc 尝试了同样的事情.我的理解是，当我传递参数 MEM_RESERVE 时，我就可以将该内存用于我想要的任何东西，例如映射文件的视图.但我发现这是不可能的(与上面相同的错误代码)，直到我再次使用 VirtualFree 完全释放整个块.因此，将不再为下一个文件保留内存.

Next I tried the same thing with VirtualAloc. My understanding was that when I pass the parameter MEM_RESERVE I would then be able to use that memory for what ever I wanted, e.g. to map a view of a file. But I found out that that is not possible (same error code as above) until i completely free the whole block with VirtualFree again. Therefor there would be no reserved memory for the next files anymore.

我已经在使用低碎片堆功能，它对我们几乎没有用.重写我们的代码以仅使用较小的文件视图目前不是一种选择.我还看了这篇文章 是否可以为多次调用 MapViewOfFileEx 而回收地址空间而不会失败?但没有发现它非常有用，并希望有其他可能性.

I'm already using the low fragmentation heap feature and it is of nearly no use to us. Rewriting our code to use only smaller views of the files is not an option at the moment. I also took a look at this post Can address space be recycled for multiple calls to MapViewOfFileEx without chance of failure? but didn't find any it very useful and was hoping for an other possibility.

你有什么建议我可以做什么或我的设计可能有什么问题吗?谢谢.

Do you have any suggestions what I can do or where my design may be wrong? Thank you.

推荐答案

好吧，MapViewOfFileEx 的文档很清楚:建议的地址用于指定文件应该同时映射多个进程中的地址.这要求地址空间区域在所有涉及的进程中可用.在用于映射的区域中不能进行其他内存分配，包括使用 VirtualAlloc"

Well, the documentation for MapViewOfFileEx is clear: "The suggested address is used to specify that a file should be mapped at the same address in multiple processes. This requires the region of address space to be available in all involved processes. No other memory allocation can take place in the region that is used for mapping, including the use of the VirtualAlloc"

低碎片堆旨在防止即使相对较小的分配失败.IE.它避免了 1 个字节的空洞，因此 2 个字节的分配将保持更长时间.按照 32 位标准，您的分配并不小.

The low fragmentation heap is intended to prevent even relatively small allocations from failing. I.e. it avoids 1 byte holes so 2 byte allocations will remain possible for longer. Your allocations are not small by 32 bits standards.

实际上，这会造成伤害.如果你真的需要它，重新实现内存映射文件.所有必要的功能都可用.使用向量异常处理程序在源代码中分页，并使用 QueryWorkingSet 确定页面是否脏.

Realistically, this is going to hurt. If you really really need it, reimplement memory mapped files. All the necessary functions are available. Use a vectored exception handler to page in the source, and use QueryWorkingSet to figure out if pages are dirty.

这篇关于将几个大文件映射到内存中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！