使用 boost::iostreams::mapped_file 时的内存使用情况 [英] Memory usage when using boost::iostreams::mapped_file

查看:45
本文介绍了使用 boost::iostreams::mapped_file 时的内存使用情况的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我在这里粘贴了一些使用 boost iostream 到 mmap & 的代码.然后写入映射文件:

I am pasting some code here which uses boost iostream to mmap & then writes to the mapped file:

typedef unordered_map<int, string> work;

    int main()
    {

            work d;
            d[0] = "a";

            boost::iostreams::mapped_file_params  params;
            params.path = "map.dat";
            params.new_file_size = 1000000000;
            params.mode = (std::ios_base::out | std::ios_base::in);
            boost::iostreams::mapped_file  mf;
            mf.open(params);
            work* w = static_cast<work*>((void*)mf.data());
            w[0] = d;
            for(int i=1; i <1000000000 ;++i)
            {
                    w->insert(std::make_pair(i, "abcdef"));
            }
            mf.close();

    }

当我在带有 8 个处理器和 16GB RAM 的 centos 6 机器上执行此操作时,我观察到以下内容:

When i executed this on my centos 6 box with 8 processors and 16GB RAM, i observed the below:

  1. 当数据被插入内存映射文件时,RES(来自top命令)不断增加,直到14GB.我的印象是,当我映射文件时,VIRT 会增加而不是 RES.那么是不是当我们写入mmap文件时,先将其写入内存然后提交到磁盘?或者是否使用了任何中间缓冲区/缓存?

  1. When the data was being inserted into the memory mapped file, RES (from top command) was increasing continuously and it reached till 14GB. I was under the impression that when i mmap a file VIRT will increase and not RES. So is it that when we write to the mmap file, first its written to the memory and then commited to the disk? Or is there any intermediate buffer/cache used?

在free"命令的帮助下,我还观察到在内存使用量达到 16GB 后,会使用缓冲区.以下是上述代码执行时不同时间的 free 命令的一些快照:

With the help of "free" command , i also observed that after the memory usage reaches 16GB, buffers are used. Here are some snapshots of free command at different times when the above code was executing:

            total       used       free     shared    buffers     cached
Mem:      16334688   10530380    5804308          0     232576    9205532
-/+ buffers/cache:    1092272   15242416
Swap:     18579448     348020   18231428

            total       used       free     shared    buffers     cached
Mem:      16334688   13594208    2740480          0     232608    9205800
-/+ buffers/cache:    4155800   12178888
Swap:     18579448     348020   18231428

            total       used       free     shared    buffers     cached
Mem:      16334688   15385944     948744          0     232648    9205808
-/+ buffers/cache:    5947488   10387200
Swap:     18579448     348020   18231428

            total       used       free     shared    buffers     cached
Mem:      16334688   16160368     174320          0     204940    4049224
-/+ buffers/cache:   11906204    4428484
Swap:     18579448     338092   18241356

            total       used       free     shared    buffers     cached
Mem:      16334688   16155160     179528          0     141584    2397820
-/+ buffers/cache:   13615756    2718932
Swap:     18579448     338092   18241356

            total       used       free     shared    buffers     cached
Mem:      16334688   16195960     138728          0       5440      17556
-/+ buffers/cache:   16172964     161724
Swap:     18579448     572052   18007396

这种行为意味着什么?

与写入内存相比,将数据写入内存映射文件需要花费大量时间.这是什么原因?

It took a lot of time to write data to memory mapped file compared to writing into memory. What is the reason for this?

当我处理大量数据时,我想使用内存映射来降低 RES 的使用率.但它似乎不是这样工作的.想要将所有数据保存在内存映射文件中,并在需要时读回.

I wanted to use memory mapping to bring down the RES usage as i deal with huge data. But it does not seem to work that way. Wanted to keep all the data in memory mapped files and read them back when required.

我是否错误地使用了内存映射?或者这就是它的行为方式?

Am I using memory mapping incorrectly? Or that's the way it behaves?

推荐答案

  1. VIRT 将立即增加(所有页面都映射到进程地址空间中).每当页面被使用时,RES 就会增加,这会导致它们被分页到物理内存中.

  1. VIRT will increase immediately (all pages are mapped into the process address space). RES will increase whenever pages are used, which causes them to be paged into the physical memory.

只要有足够的可用内存,就会发生这种情况,之后操作系统开始从保留集中清除 LRU 页(除非它们是 VirtualLock/mlock-ed或者不可移动(如内核页面、DMA 缓冲区、安全敏感数据等).

This happens for as long as there is sufficient memory available, after which the OS starts purging LRU pages from the reserved sets (unless they were VirtualLock/mlock-ed or are otherwise unmovable (like kernel pages, DMA buffers, security sensitive data etc.).

因此,操作系统乐观地保留尽可能长的页面(只要没有其他进程争用内存,就可以提高性能).

So, the OS optimistically leaves the pages reserved as long as possible (which improves performance as long as no other processes contend for the memory).

这表示操作系统正在完成它的工作.

This signifies that the OS is doing it's job.

您正在写入磁盘.磁盘访问比内存访问慢(很多).数据实际写入磁盘的频率取决于调整.此答案列出了 linux 上可用的一些调整参数(您似乎正在使用):

You're writing to disk. Disk access is (a lot) slower than memory access. How often the data actually gets written out to disk depends on tuning. This answer lists some of the tuning parameters that are available on linux (which you seem to be using):

这篇关于使用 boost::iostreams::mapped_file 时的内存使用情况的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆