使用页面文件进行缓存？ [英] Using the pagefile for caching?

问题描述

我必须处理大量的数据，通常不适合主内存。我访问这个数据的方式具有高局部性，因此在内存中缓存它的部分是一个很好的选择。是否可以只使用malloc（）一个巨大的数组，并让操作系统找出哪些位要翻页和保留哪些位？

I have to deal with a huge amount of data that usually doesn't fit into main memory. The way I access this data has high locality, so caching parts of it in memory looks like a good option. Is it feasible to just malloc() a huge array, and let the operating system figure out which bits to page out and which bits to keep?

推荐答案

假设数据来自一个文件，你最好是内存映射该文件。否则，你最后要做的是分配数组，然后将数据从文件复制到数组 - 并且由于数组被映射到页面文件，你基本上只是将原始文件复制到页面文件，并且在该过程中污染缓存（即，物理存储器），使得当前活动的其他数据具有更好的被逐出的机会。然后，完成后（通常）将数据从数组写回原始文件，这意味着将页面文件复制回原始文件。

Assuming the data comes from a file, you're better off memory mapping that file. Otherwise, what you end up doing is allocating your array, and then copying the data from your file into the array -- and since your array is mapped to the page file, you're basically just copying the original file to the page file, and in the process polluting the "cache" (i.e., physical memory) so other data that's currently active has a much better chance of being evicted. Then, when you're done you (typically) write the data back from the array to the original file, which (in this case) means copying from the page file back to the original file.

内存映射文件改为只是创建一些地址空间，并将其直接映射到原始文件。这避免了将数据从原始文件复制到页面文件（并在完成后再次复制），以及在从原始文件到页面文件的路径上将数据临时移动到物理内存中。最大的胜利，当然是，当/如果原始文件的实质部分，你从来没有真正使用（在这种情况下，他们可能永远不会被读入物理内存中，假设未使用的块至少一个页面尺寸）。

Memory mapping the file instead just creates some address space and maps it directly to the original file instead. This avoids copying data from the original file to the page file (and back again when you're done) as well as temporarily moving data into physical memory on the way from the original file to the page file. The biggest win, of course, is when/if there are substantial pieces of the original file that you never really use at all (in which case they may never be read into physical memory at all, assuming the unused chunk is at least a page in size).

这篇关于使用页面文件进行缓存？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！