使用boost ::时，内存使用量的iostream :: mapped_file所 [英] Memory usage when using boost::iostreams::mapped_file
问题描述
我粘贴一些$ C $其中，C，它使用的iostream提振对mmap＆安培;然后写入映射文件：
的typedef unordered_map＆LT; INT，串＆GT;工作; 诠释的main（）
{ 工作D组;
D [0] =A; 提高::输入输出流:: mapped_file_params PARAMS;
params.path =map.dat;
params.new_file_size = 10亿;
params.mode =（的std :: ::的ios_base出来|的std :: ::的ios_base中）;
提高::输入输出流:: mapped_file所MF;
mf.open（PARAMS）;
工作* W =的static_cast＆LT;工作* GT;（（无效*）mf.data（））;
W [0] = d的;
的for（int i = 1; I＆LT; 10亿; ++ I）
{
W-＆GT;插入（的std :: make_pair（我，ABCDEF））;
}
mf.close（）; }
当我执行该对我的CentOS 6盒8个处理器和16GB的RAM，我观察到如下：
-
当数据被插入存储器映射文件，RES（从顶部命令）连续增加，则达到直到14GB。
我是IM pression，当我的mmap文件VIRT会增加，而不是RES之下。
因此，它是当我们写MMAP文件中，首次写入内存，然后将提交给磁盘？或者有使用的任何中间缓冲区/缓存？ -
通过免费命令的帮助，我也观察到，内存使用量达到16GB后，使用缓冲区。以下是在不同的时间free命令的一些快照时上述code的执行：
共免费使用共享缓冲区高速缓存
纪念品：16334688 10530380 5804308 0 232576 9205532
- / +缓冲区/缓存：1092272 15242416
交换：18579448 348020 18231428 共免费使用共享缓冲区高速缓存
纪念品：16334688 13594208 2740480 0 232608 9205800
- / +缓冲区/缓存：4155800 12178888
交换：18579448 348020 18231428 共免费使用共享缓冲区高速缓存
纪念品：16334688 15385944 948744 0 232648 9205808
- / +缓冲区/缓存：5947488 10387200
交换：18579448 348020 18231428 共免费使用共享缓冲区高速缓存
纪念品：16334688 16160368 174320 0 204940 4049224
- / +缓冲区/缓存：11906204 4428484
交换：18579448 338092 18241356 共免费使用共享缓冲区高速缓存
纪念品：16334688 16155160 179528 0 141584 2397820
- / +缓冲区/缓存：13615756 2718932
交换：18579448 338092 18241356 共免费使用共享缓冲区高速缓存
纪念品：16334688 16195960 138728 0 5440 17556
- / +缓冲区/缓存：16172964 161724
交换：18579448 572052 18007396这是什么行为意味着什么？
-
我花了很多时间相比，写入到内存中写入数据到内存映射文件。是什么原因？
我想使用内存映射打倒RES用法我处理庞大的数据。但它似乎并没有这样的。想保持在内存映射文件中的所有数据，并在需要时阅读他们回来。
我使用内存映射不正确？或者，这是它的行为方式？
-
VIRT将立即增加（所有页面的映射的到进程的地址空间）。 RES将增加每当页面的使用的，这将导致它们被分页到物理内存。
这会发生，只要有足够的可用内存，在此之后，操作系统开始从保留台清洗LRU页面（除非它们是
VirtualLock/MLOCK-ed或在其他不可移动（如内核页，DMA缓冲区，安全敏感数据等）。因此，OS乐观叶保留页面尽可能长的时间（其中，只要没有其他进程争提高性能的内存）。
-
这意味着操作系统是这样做的工作。
-
您正在写入磁盘。磁盘访问（很多），比内存访问速度慢。多久实际得到的数据写入到磁盘取决于调整。这个答案列出了一些调整参数可用在Linux（你似乎可以用）的：
I am pasting some code here which uses boost iostream to mmap & then writes to the mapped file:
typedef unordered_map<int, string> work;
int main()
{
work d;
d[0] = "a";
boost::iostreams::mapped_file_params params;
params.path = "map.dat";
params.new_file_size = 1000000000;
params.mode = (std::ios_base::out | std::ios_base::in);
boost::iostreams::mapped_file mf;
mf.open(params);
work* w = static_cast<work*>((void*)mf.data());
w[0] = d;
for(int i=1; i <1000000000 ;++i)
{
w->insert(std::make_pair(i, "abcdef"));
}
mf.close();
}
When i executed this on my centos 6 box with 8 processors and 16GB RAM, i observed the below:
When the data was being inserted into the memory mapped file, RES (from top command) was increasing continuously and it reached till 14GB. I was under the impression that when i mmap a file VIRT will increase and not RES. So is it that when we write to the mmap file, first its written to the memory and then commited to the disk? Or is there any intermediate buffer/cache used?
With the help of "free" command , i also observed that after the memory usage reaches 16GB, buffers are used. Here are some snapshots of free command at different times when the above code was executing:
total used free shared buffers cached Mem: 16334688 10530380 5804308 0 232576 9205532 -/+ buffers/cache: 1092272 15242416 Swap: 18579448 348020 18231428 total used free shared buffers cached Mem: 16334688 13594208 2740480 0 232608 9205800 -/+ buffers/cache: 4155800 12178888 Swap: 18579448 348020 18231428 total used free shared buffers cached Mem: 16334688 15385944 948744 0 232648 9205808 -/+ buffers/cache: 5947488 10387200 Swap: 18579448 348020 18231428 total used free shared buffers cached Mem: 16334688 16160368 174320 0 204940 4049224 -/+ buffers/cache: 11906204 4428484 Swap: 18579448 338092 18241356 total used free shared buffers cached Mem: 16334688 16155160 179528 0 141584 2397820 -/+ buffers/cache: 13615756 2718932 Swap: 18579448 338092 18241356 total used free shared buffers cached Mem: 16334688 16195960 138728 0 5440 17556 -/+ buffers/cache: 16172964 161724 Swap: 18579448 572052 18007396What does this behavior signify?
It took a lot of time to write data to memory mapped file compared to writing into memory. What is the reason for this?
I wanted to use memory mapping to bring down the RES usage as i deal with huge data. But it does not seem to work that way. Wanted to keep all the data in memory mapped files and read them back when required.
Am I using memory mapping incorrectly? Or that's the way it behaves?
VIRT will increase immediately (all pages are mapped into the process address space). RES will increase whenever pages are used, which causes them to be paged into the physical memory.
This happens for as long as there is sufficient memory available, after which the OS starts purging LRU pages from the reserved sets (unless they were
VirtualLock/mlock-ed or are otherwise unmovable (like kernel pages, DMA buffers, security sensitive data etc.).So, the OS optimistically leaves the pages reserved as long as possible (which improves performance as long as no other processes contend for the memory).
This signifies that the OS is doing it's job.
You're writing to disk. Disk access is (a lot) slower than memory access. How often the data actually gets written out to disk depends on tuning. This answer lists some of the tuning parameters that are available on linux (which you seem to be using):
这篇关于使用boost ::时，内存使用量的iostream :: mapped_file所的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
