使用 boost::iostreams::mapped_file 时的内存使用情况 [英] Memory usage when using boost::iostreams::mapped_file
问题描述
我在这里粘贴了一些使用 boost iostream 到 mmap & 的代码.然后写入映射文件:
I am pasting some code here which uses boost iostream to mmap & then writes to the mapped file:
typedef unordered_map<int, string> work;
int main()
{
work d;
d[0] = "a";
boost::iostreams::mapped_file_params params;
params.path = "map.dat";
params.new_file_size = 1000000000;
params.mode = (std::ios_base::out | std::ios_base::in);
boost::iostreams::mapped_file mf;
mf.open(params);
work* w = static_cast<work*>((void*)mf.data());
w[0] = d;
for(int i=1; i <1000000000 ;++i)
{
w->insert(std::make_pair(i, "abcdef"));
}
mf.close();
}
当我在带有 8 个处理器和 16GB RAM 的 centos 6 机器上执行此操作时,我观察到以下内容:
When i executed this on my centos 6 box with 8 processors and 16GB RAM, i observed the below:
当数据被插入内存映射文件时,RES(来自top命令)不断增加,直到14GB.我的印象是,当我映射文件时,VIRT 会增加而不是 RES.那么是不是当我们写入mmap文件时,先将其写入内存然后提交到磁盘?或者是否使用了任何中间缓冲区/缓存?
When the data was being inserted into the memory mapped file, RES (from top command) was increasing continuously and it reached till 14GB. I was under the impression that when i mmap a file VIRT will increase and not RES. So is it that when we write to the mmap file, first its written to the memory and then commited to the disk? Or is there any intermediate buffer/cache used?
在free"命令的帮助下,我还观察到在内存使用量达到 16GB 后,会使用缓冲区.以下是上述代码执行时不同时间的 free 命令的一些快照:
With the help of "free" command , i also observed that after the memory usage reaches 16GB, buffers are used. Here are some snapshots of free command at different times when the above code was executing:
total used free shared buffers cached
Mem: 16334688 10530380 5804308 0 232576 9205532
-/+ buffers/cache: 1092272 15242416
Swap: 18579448 348020 18231428
total used free shared buffers cached
Mem: 16334688 13594208 2740480 0 232608 9205800
-/+ buffers/cache: 4155800 12178888
Swap: 18579448 348020 18231428
total used free shared buffers cached
Mem: 16334688 15385944 948744 0 232648 9205808
-/+ buffers/cache: 5947488 10387200
Swap: 18579448 348020 18231428
total used free shared buffers cached
Mem: 16334688 16160368 174320 0 204940 4049224
-/+ buffers/cache: 11906204 4428484
Swap: 18579448 338092 18241356
total used free shared buffers cached
Mem: 16334688 16155160 179528 0 141584 2397820
-/+ buffers/cache: 13615756 2718932
Swap: 18579448 338092 18241356
total used free shared buffers cached
Mem: 16334688 16195960 138728 0 5440 17556
-/+ buffers/cache: 16172964 161724
Swap: 18579448 572052 18007396
这种行为意味着什么?
与写入内存相比,将数据写入内存映射文件需要花费大量时间.这是什么原因?
It took a lot of time to write data to memory mapped file compared to writing into memory. What is the reason for this?
当我处理大量数据时,我想使用内存映射来降低 RES 的使用率.但它似乎不是这样工作的.想要将所有数据保存在内存映射文件中,并在需要时读回.
I wanted to use memory mapping to bring down the RES usage as i deal with huge data. But it does not seem to work that way. Wanted to keep all the data in memory mapped files and read them back when required.
我是否错误地使用了内存映射?或者这就是它的行为方式?
Am I using memory mapping incorrectly? Or that's the way it behaves?
推荐答案
VIRT 将立即增加(所有页面都映射到进程地址空间中).每当页面被使用时,RES 就会增加,这会导致它们被分页到物理内存中.
VIRT will increase immediately (all pages are mapped into the process address space). RES will increase whenever pages are used, which causes them to be paged into the physical memory.
只要有足够的可用内存,就会发生这种情况,之后操作系统开始从保留集中清除 LRU 页(除非它们是 VirtualLock
/mlock
-ed或者不可移动(如内核页面、DMA 缓冲区、安全敏感数据等).
This happens for as long as there is sufficient memory available, after which the OS starts purging LRU pages from the reserved sets (unless they were VirtualLock
/mlock
-ed or are otherwise unmovable (like kernel pages, DMA buffers, security sensitive data etc.).
因此,操作系统乐观地保留尽可能长的页面(只要没有其他进程争用内存,就可以提高性能).
So, the OS optimistically leaves the pages reserved as long as possible (which improves performance as long as no other processes contend for the memory).
这表示操作系统正在完成它的工作.
This signifies that the OS is doing it's job.
您正在写入磁盘.磁盘访问比内存访问慢(很多).数据实际写入磁盘的频率取决于调整.此答案列出了 linux 上可用的一些调整参数(您似乎正在使用):
You're writing to disk. Disk access is (a lot) slower than memory access. How often the data actually gets written out to disk depends on tuning. This answer lists some of the tuning parameters that are available on linux (which you seem to be using):
这篇关于使用 boost::iostreams::mapped_file 时的内存使用情况的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!