序列化过程中的boost ::系列化高的内存消耗 [英] boost::serialization high memory consumption during serialization

问题描述

结果
正如题目暗示序列化大量的数据到一个文件时，我遇到一个轻微的问题与升压::序列化。这个问题是由服用约3至3.5倍我的对象的内存被序列化的应用程序序列化部分的内存占用。结果
需要注意的是数据结构我已经是基类的指针的三维向量和指向该结构是重要的。像这样的：

just as the topic suggests I've come across a slight issue with boost::serialization when serializing a huge amount of data to a file. The problem consists of the memory footprint of the serialization part of the application taking around 3 to 3.5 times the memory of my objects being serialized.
It is important to note that the data structure I have is a three dimensional vector of base class pointers and a pointer to that structure. Like this:

using namespace std;    
vector<vector<vector<MyBase*> > >* data;

这是后来以code模拟这一个序列：

This is later serialised with a code analog to this one:

ar & BOOST_SERIALIZATION_NVP(data);

升压/系列化/ vector.hpp也包括在内。

boost/serialization/vector.hpp is included.

班正在连载全部来自MyBase继承。结果
现在，因为我的项目开始我用不同的档案馆从典型binary_archive，文本，XML和最后多态性二元/ XML /文本序列化。这些行为每一个人完全相同的方式。搜索结果
通常情况下，这将不会是一个问题，如果我有序列化少量的数据，但我有类的数量都在milions（最好是10左右畅想）和我已经能够测试它的内存使用情况显示的一致通过提振::在code系列化部分分配的内存是围绕整个应用程序的内存占用的2/3，而写文件。搜索结果
这相当于13.5左右GB的采取4畅想对象所在的对象本身4.2GB RAM需要。现在，这是据我已经能够把我的code，因为我没有获得一台机器比物理内存8GB以上。我还要指出，这是一个64位的应用程序正在在Windows 7 Professional x64 Edition的运行，但形势的Ubuntu盒相似。搜索结果
任何人有任何想法，我怎么会去有关解决这个，因为它是不能接受的我有，同时运行，因为它确实序列化时，将不使用尽可能多的内存的应用程序如此高的内存需求。搜索结果
反序列化是不是那么糟糕，因为它分配约1.5倍，需要的内存。这是我能够接受。

Classes being serialised all inherit from "MyBase".
Now, since the start of my project I've used different archives for serialization from typical binary_archive, text, xml and finally polymorphic binary/xml/text. Every single one of these acts exactly the same way.

Typically this wouldn't be a problem if I had to serialize small amounts of data but the number of classes I have are in the milions (ideally around 10 milion) and the memory usage as I've been able to test it shows consistently that the memory allocated by boost::serialization part of the code is around 2/3 of the application whole memory footprint while writing the file.

This amounts to around 13.5 GB of RAM taken for 4 milion objects where the objects themselves take 4.2GB. Now this is as far as I've been able to take my code since I don't have access to a machine with more than 8GB of physical RAM. I should also note that this is a 64bit application being run on a Windows 7 professional x64 edition but the situation is similar on an Ubuntu box.

Anyone has any idea how I would go about troubleshooting this as it is unacceptable for me to have such high memory requirements for an application that will not use as much memory while running as it does while serializing.

Deserialization isn't as bad, as it allocates around 1.5 times the needed memory. This is something I could live with.

试图转向跟踪了与升压::档案:: archive_flags :: no_tracking，但它的作用完全一样的。搜索结果
任何人有任何想法，我应该怎么办呢？

Tried turning tracking off with boost::archive::archive_flags::no_tracking but it acts exactly the same.

Anyone have any idea what I should do?

推荐答案

使用的valgrind我发现，内存消耗的主要原因是库跟踪指针里面的地图。如果你确信你不需要指针跟踪（这意味着你确信没有指针别名）禁用跟踪。你可以找到这里的主要概念的禁用跟踪。总之，你必须做这样的事情：

Using valgrind I found that the main reason of memory consumption is a map inside the library to track pointers. If you are certain that you do not need pointer tracking ( it means you are sure that there is no pointer aliasing) disable tracking. You can find here the main concepts of disable tracking. In short you must do something like this:

BOOST_CLASS_TRACKING(vector<vector<vector<MyBase*> > >, boost::serialization::track_never)

在<一个href=\"http://stackoverflow.com/questions/35391792/set-tracking-traits-of-template-class-in-boost-serialization-to-reduce-memory-co\">my问题我写了一个版本的宏，您可以禁用模板类的跟踪。这必须在你的内存消耗显著的影响。
还要注意，有任何容器内的指针如果你想跟踪从未必须禁止它们跟踪了。目前，我找不到任何方法来正确做到这一点。

In my question I wrote a version of this macro that you could disable tracking of a template class. This must have a significant impact on your memory consumption. Also notice that there are pointers inside any containers If you want tracking never you must disable tracking of them too. Currently I could not find any way to do this properly.

这篇关于序列化过程中的boost ::系列化高的内存消耗的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！