内存映射文件，受管映射文件和偏移指针 [英] Memory Mapped Files, Managed Mapped File and Offset Pointer

问题描述

我对Boost库（对于Windows）的术语有点困惑。我想做的只是：在磁盘上创建文件（大文件> 50 GB）分别为写入和读取操作执行一些映射。

I'm a little bit confused about the terminology of Boost Library (for windows). What I'm trying to do is simply; create a file on disk (a big file >50 GB) do some mapping for write and read operations seperately.

例如，首先映射1 gb部分用于写入&然后刷新硬盘驱动器采取一个新的部分，依此类推，而读者应用程序映射文件的不同部分，并做阅读的东西，而不改变任何东西（无编辑）。

For example first map 1 gb portion for writing & after that flush it to the hard drive take a new portion and so on, while the reader applications maps different parts of the file and do the reading stuff without changing anything (no edit).

我正在阅读boost的文档（1.47.0版本，因为我们允许使用这一个），我不知道什么时候使用内存映射文件方法，如：file_mapping， managed_region和管理地图文件：basic_managed_mapped_file和Offset_Ptr。

I'm reading the documentation of boost (1.47.0 version since we allowed to use this one) and I don't understand exactly when to use Memory Mapped Files methods like: file_mapping, managed_region and Managed Map File: basic_managed_mapped_file and Offset_Ptr for instance.

任何人都可以告诉我内存映射文件和管理映射文件和它们的用途是什么？

Can anyone please tell me what is the difference between Memory Mapped Files and Managed Mapped File and what are the usages of them?

如果可能的话，一些示例代码将对这些和Offset_ptr高度解析。

Some example codes would be highly apopriciated about these and Offset_ptr as well if possible.

..

推荐答案

您可以使用 managed_mapped_file 内存映射文件。

You can use the managed_mapped_file to transparently allocate from a memory mapped file.

这意味着对于所有实际用途，您通常不需要重新分配内存区域。它是所有的虚拟内存无论如何，所以分页需要在所需的时间加载正确的位。

This means that for all practical purposes you often don't need to dubdivide your memory areas. It's all virtual memory anyways, so paging takes care of loading the right bits at the required times.

显然，如果有很多碎片或访问跳跃，则分页可能会成为性能瓶颈。在这种情况下，考虑细分为池并从中分配。）_

Obviously, if there's a lot of fragmentation or accesses "jumping around" then paging might become a performance bottleneck. In that case, consider subdividing into pools and allocate from those.)_

编辑只需注意Boost IPC支持< =http://www.boost.org/doc/libs/1_55_0/doc/html/interprocess/allocators_containers.html#interprocess.allocators_containers.stl_allocators_segregated_storage =nofollow>隔离的存储节点分配器和< a href =http://www.boost.org/doc/libs/1_55_0/doc/html/interprocess/allocators_containers.html#interprocess.allocators_containers.stl_allocators_adaptive =nofollow>自适应池节点分配器 。还有关于这些存储池的实施的注释

Edit Just noticed Boost IPC has support for this under Segregated storage node allocators and Adaptive pool node allocators. There are also notes about the implementation of these storage pools here.

这里有一个简单的起点，创建一个50Gb文件并在其中填入一些数据：

Here's a simple starting point that creates a 50Gb file and stuffs some data in it:

#include <iostream>
#include <string>
#include <vector>
#include <iterator>
#include <algorithm>

#include <boost/container/flat_map.hpp>
#include <boost/container/flat_set.hpp>

#include <boost/interprocess/managed_mapped_file.hpp>
#include <boost/container/scoped_allocator.hpp>
#include <boost/interprocess/containers/string.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <boost/interprocess/sync/named_mutex.hpp>
#include <boost/interprocess/sync/scoped_lock.hpp>

namespace bip = boost::interprocess;
using mutex_type    = bip::named_mutex;

struct X
{
    char buf[100];
    double rate;
    uint32_t samples[1024];
};

template <typename T> using shared_alloc  = bip::allocator<T,bip::managed_mapped_file::segment_manager>;
template <typename T> using shared_vector = boost::container::vector<T, shared_alloc<T> >;
template <typename K, typename V, typename P = std::pair<K,V>, typename Cmp = std::less<K> >
                      using shared_map    = boost::container::flat_map<K, V, Cmp, shared_alloc<P> >;

using shared_string = bip::basic_string<char,std::char_traits<char>,shared_alloc<char> >;
using dataset_t     = shared_map<shared_string, shared_vector<X> >;

struct mutex_remove
{
    mutex_remove() { mutex_type::remove("7FD6D7E8-320B-11DC-82CF-39598D556B0E"); }
    ~mutex_remove(){ mutex_type::remove("7FD6D7E8-320B-11DC-82CF-39598D556B0E"); }
} remover;

static mutex_type mutex(bip::open_or_create,"7FD6D7E8-320B-11DC-82CF-39598D556B0E");

static dataset_t& shared_instance()
{
    bip::scoped_lock<mutex_type> lock(mutex);
    static bip::managed_mapped_file seg(bip::open_or_create,"./demo.db", 50ul<<30); // "50Gb ought to be enough for anyone"

    static dataset_t* _instance = seg.find_or_construct<dataset_t>
        ("DATA")
        (
         std::less<shared_string>(), 
         dataset_t::allocator_type(seg.get_segment_manager())
        );

    static auto capacity = seg.get_free_memory();
    std::cerr << "Free space: " << (capacity>>30) << "g\n";

    return *_instance;
}

int main()
{
    auto& db = shared_instance();

    bip::scoped_lock<mutex_type> lock(mutex);
    auto alloc = db.get_allocator().get_segment_manager();

    std::cout << db.size() << '\n';

    for (int i = 0; i < 1000; ++i)
    {
        std::string key_ = "item" + std::to_string(i);
        shared_string key(alloc);
        key.assign(key_.begin(), key_.end());
        auto value = shared_vector<X>(alloc);
        value.resize(size_t(rand()%(1ul<<9)));
        auto entry = std::make_pair(key, value);

        db.insert(std::make_pair(key, value));
    }
}

注意它写了一个文件50G。提交的实际大小取决于随机的一点。我的运行导致大约1.1G：

Note that it writes a sparse file of 50G. The actual size commited depends on a bit of random there. My run resulted in a roughly 1.1G:

$ du -shc --apparent-size demo.db 
50G demo.db

$ du -shc demo.db 
1,1G    demo.db

希望这有助于

这篇关于内存映射文件，受管映射文件和偏移指针的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！