内存映射文件，管理映射文件和偏移指针 [英] Memory Mapped Files, Managed Mapped File and Offset Pointer

问题描述

我有点困惑Boost库中的术语（适用于Windows）。我试图做的是简单的;在磁盘上创建一个文件（大文件> 50 GB）做写一些映射和单独读取操作。

I'm a little bit confused about the terminology of Boost Library (for windows). What I'm trying to do is simply; create a file on disk (a big file >50 GB) do some mapping for write and read operations seperately.

有关写入和放大器例如第一张地图1 GB的部分;这刷新它的硬盘后，采取新的部分等等，而阅读器应用程序映射文件的不同部分，做阅读的东西不改变任何东西（没有编辑）。

For example first map 1 gb portion for writing & after that flush it to the hard drive take a new portion and so on, while the reader applications maps different parts of the file and do the reading stuff without changing anything (no edit).

我在读提振的文件（因为我们允许使用这一个1.47.0版本），我不完全了解什么时候使用内存映射文件类似的方法：file_mapping，managed_region和管理地图文件：basic_managed_mapped_file和Offset_Ptr例如

I'm reading the documentation of boost (1.47.0 version since we allowed to use this one) and I don't understand exactly when to use Memory Mapped Files methods like: file_mapping, managed_region and Managed Map File: basic_managed_mapped_file and Offset_Ptr for instance.

谁能告诉我什么是内存之间的映射文件和托管映射文件的区别，什么是他们的惯例？

Can anyone please tell me what is the difference between Memory Mapped Files and Managed Mapped File and what are the usages of them?

一些示例codeS将高度apopriciated关于这些和Offset_ptr以及如果可能的话。

Some example codes would be highly apopriciated about these and Offset_ptr as well if possible.

感谢您确实...

推荐答案

您可以使用 managed_mapped_file 来透明地从内存映射文件分配。

You can use the managed_mapped_file to transparently allocate from a memory mapped file.

这意味着对于所有的实际目的你常常不需要dubdivide您的内存区域。这是所有的虚拟内存反正，所以分页需要在规定的时间加载正确的位的照顾。

This means that for all practical purposes you often don't need to dubdivide your memory areas. It's all virtual memory anyways, so paging takes care of loading the right bits at the required times.

显然，如果有很多碎片或者访问跳来跳去，然后页面可能成为性能瓶颈。在这种情况下，考虑细分成池，并从这些分配。）_

Obviously, if there's a lot of fragmentation or accesses "jumping around" then paging might become a performance bottleneck. In that case, consider subdividing into pools and allocate from those.)_

修改只注意到升压IPC已根据<一个这种支持href=\"http://www.boost.org/doc/libs/1_55_0/doc/html/interprocess/allocators_containers.html#interprocess.allocators_containers.stl_allocators_segregated_storage\"相对=nofollow>种族隔离存储节点分配器和<一个href=\"http://www.boost.org/doc/libs/1_55_0/doc/html/interprocess/allocators_containers.html#interprocess.allocators_containers.stl_allocators_adaptive\"相对=nofollow>自适应池节点分配器。也有说明有关这些存储池的实施<一个href=\"http://www.boost.org/doc/libs/1_55_0/doc/html/interprocess/architecture.html#interprocess.architecture.allocators_containers.implementation_segregated_storage_pools\"相对=nofollow>这里。

Edit Just noticed Boost IPC has support for this under Segregated storage node allocators and Adaptive pool node allocators. There are also notes about the implementation of these storage pools here.

下面是创建一个50GB的文件，并在它填充一些数据的简单的出发点：

Here's a simple starting point that creates a 50Gb file and stuffs some data in it:

#include <iostream>
#include <string>
#include <vector>
#include <iterator>
#include <algorithm>

#include <boost/container/flat_map.hpp>
#include <boost/container/flat_set.hpp>

#include <boost/interprocess/managed_mapped_file.hpp>
#include <boost/container/scoped_allocator.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/interprocess/sync/named_mutex.hpp>
#include <boost/interprocess/sync/scoped_lock.hpp>

namespace bip = boost::interprocess;
using mutex_type    = bip::named_mutex;

struct X
{
    char buf[100];
    double rate;
    uint32_t samples[1024];
};

template <typename T> using shared_alloc  = bip::allocator<T,bip::managed_mapped_file::segment_manager>;
template <typename T> using shared_vector = boost::container::vector<T, shared_alloc<T> >;
template <typename K, typename V, typename P = std::pair<K,V>, typename Cmp = std::less<K> >
                      using shared_map    = boost::container::flat_map<K, V, Cmp, shared_alloc<P> >;

using shared_string = bip::basic_string<char,std::char_traits<char>,shared_alloc<char> >;
using dataset_t     = shared_map<shared_string, shared_vector<X> >;

struct mutex_remove
{
    mutex_remove() { mutex_type::remove("7FD6D7E8-320B-11DC-82CF-39598D556B0E"); }
    ~mutex_remove(){ mutex_type::remove("7FD6D7E8-320B-11DC-82CF-39598D556B0E"); }
} remover;

static mutex_type mutex(bip::open_or_create,"7FD6D7E8-320B-11DC-82CF-39598D556B0E");

static dataset_t& shared_instance()
{
    bip::scoped_lock<mutex_type> lock(mutex);
    static bip::managed_mapped_file seg(bip::open_or_create,"./demo.db", 50ul<<30); // "50Gb ought to be enough for anyone"

    static dataset_t* _instance = seg.find_or_construct<dataset_t>
        ("DATA")
        (
         std::less<shared_string>(), 
         dataset_t::allocator_type(seg.get_segment_manager())
        );

    static auto capacity = seg.get_free_memory();
    std::cerr << "Free space: " << (capacity>>30) << "g\n";

    return *_instance;
}

int main()
{
    auto& db = shared_instance();

    bip::scoped_lock<mutex_type> lock(mutex);
    auto alloc = db.get_allocator().get_segment_manager();

    std::cout << db.size() << '\n';

    for (int i = 0; i < 1000; ++i)
    {
        std::string key_ = "item" + std::to_string(i);
        shared_string key(alloc);
        key.assign(key_.begin(), key_.end());
        auto value = shared_vector<X>(alloc);
        value.resize(size_t(rand()%(1ul<<9)));
        auto entry = std::make_pair(key, value);

        db.insert(std::make_pair(key, value));
    }
}

请注意，它写入的稀疏的50G的文件。 COMMITED实际大小取决于比特的随机出现。我的运行造成了大约1.1G：

Note that it writes a sparse file of 50G. The actual size commited depends on a bit of random there. My run resulted in a roughly 1.1G:

$ du -shc --apparent-size demo.db 
50G demo.db

$ du -shc demo.db 
1,1G    demo.db

希望这有助于

这篇关于内存映射文件，管理映射文件和偏移指针的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！