未分配堆没有回收？ [英] Deallocated Heap Not Being Reclaimed?

问题描述

我有一个C ++进程，它获取大块数据并将它们存储在内存中。存储阵列包含大约10 GB的数据，分区为4MB块。当新数据到达时，它创建一个新块，然后如果它已满，则删除旧块。此过程每10 - 60秒循环一次完整的循环缓冲区。我们在x86_64 RH5和RH6上运行，并使用Intel 14编译器进行编译。

我们看到一个问题，即整个进程内存使用情况随着时间的推移而增长，直到操作系统运行内存不足，最终盒子死亡。我们一直在寻找内存泄漏，并通过TotalView运行过程，试图确定内存在哪里并且没有看到任何报告的泄漏。

在堆报告总视图我们看到了为存储数据分配的10GB内存，但我们也看到了4+ GB的释放内存。通过堆显示，看来我们的堆非常碎片。

• 是已解除分配的内存。 已经被我的进程释放但是没有被操作系统回收的内存内存，并且认为这可能是我们的内存泄漏的源头是合理的吗？




• 如果是，我如何让操作系统回收内存？




• 我们需要重做我们的过程，在操作系统上为我们进行内存管理？

解决方案（和希望你），你在Linux上（如果将代码移植到Linux是可行的，考虑到因为Linux有这样的问题的好工具）。

然后：

• 使用C ++ 11（或C ++ 14），并了解移动语义，智能指针和规则五




• 使用 valgrind

  / li>
  

• 使用您最近 GCC / a>或 Clang / LLVM 编译器。请参阅 -fsanitizer = 。 ..调试选项;您至少在调试期间可能需要 -fsanitize = address 。

以上内容将帮助您捕获一些剩余的内存泄漏。准备好花几个星期。您可能需要停用 ASLR ，您应该了解 gdb 观察点。

您也可以考虑使用 Boehm的保守的垃圾回收器。请参阅此，以便在标准C ++中使用容器。如果你使用Boehm的GC，你最好在程序的每一个地方使用它...

正版碎片可能会发生（即使你确定已经避免了内存泄漏，并检查了例如 valgrind ），特别是对于长期进程。在这种情况下，您可以考虑使用自己的应用程序检查点功能（这对重新启动长期计算）。如果你早点考虑了（检查点应该是一个早期的架构设计决策！）你可以检查点你的状态到磁盘一段时间（例如每小时），并重新启动一个新的过程。这可以是一个很好的内存压缩策略。

你可以（但我不一定建议）在OS上写自己的内存分配器虚拟地址空间更改原型，例如 mmap（2）（可能有 MAP_HUGETLB ....）& munmap ;你可能有自己的allocator和deallocator（至少对于大型对象，或者 operator new & operator delete ，等等，在的一些类），阅读关于C ++ 分配器概念。但你的标准 new 和 delete （和 malloc & 免费用于C代码，通常由C ++使用 new & delete



请注意，大多数免费或 delete 不调用 munmap ，而是简单地将释放的内存标记为可由未来 malloc 或 new ...

你肯定应该更熟悉垃圾收集技术和术语。阅读 GC手​​册。

另请参阅 mallinfo（3）& mallopt（3）& proc（5）（可能使用 / proc / self / maps 或 / proc / self / smaps & / proc / self / statm 从程序内部了解你的堆，或 pmap 命令）。也许 strace（1）可能有用（了解什么< a href =http://man7.org/linux/man-pages/man2/syscalls.2.html =nofollow> syscalls（2）发生）

I have C++ process that ingests large blocks of data and stores them in memory. The storage array contains roughly 10 GB of data partitioned into 4MB blocks. As new data arrives it creates a new block and then deletes an old block if it is full. This process cycles through the full circular buffer once every 10 - 60 seconds. We are running on an x86_64 RH5 and RH6 and compiling with the Intel 14 compiler.

We are seeing a problem where the overall process memory usage grows over time until the OS runs out of memory and eventually the box dies. We have been looking for memory leaks and running the process through TotalView trying to determine where the memory is going and are not seeing any reported leaks.

On the heap report produced by total view we saw the 10GB of allocated memory for the stored data, but we also saw 4+ GB of "deallocated" memory. Looking through the heap display, it appeared that our heap was very fragmented. There would be a large chunk of "allocated" memory interspersed with large chunks of "deallocated" memory.

• Is the "deallocated" memory memory that has been freed by my process but not reclaimed by the OS and is it reasonable to think that this may be the source of our memory "leak"?

• If so, how do I get the OS to reclaim the memory?

• Do we need to rework our process to reuse discarded data blocks instead of relying on the OS to do our memory management for us?

解决方案

I guess (and hope for you) that you are on Linux (if porting your code to Linux is doable, consider that since Linux has good tools for such issues).

Then:

• use C++11 (or C++14) and learn about move semantics, smart pointers, and rule of five.

• use valgrind

• use some sanitizers from your recent GCC or Clang/LLVM compiler. Read about -fsanitizer=... debugging options; you probably want -fsanitize=address at least during debugging.

The above will help you catching some remaining memory leaks. Be prepared to spend weeks on them. You might need to disable ASLR and you should learn about gdb watchpoints.

You might also consider using Boehm's conservative garbage collector. See this for using it in standard C++ containers. If you do use Boehm's GC you'll better use it nearly every where in your program ...

Genuine fragmentation may happen (even if you are sure to have avoided memory leaks, and have checked that e.g. with valgrind), in particular for long lived processes. In such cases, you might consider having your own application checkpointing facilities (which are also useful to restart a long-lived computation). If you have thought about it early enough (checkpointing should be an early architectural design decision!) you could checkpoint your state to disk once in a while (e.g. every hour) and restart a fresh process. This can be a good memory compacting strategy.

You could (but I don't necessarily recommend) writing your own memory allocator above OS virtual address space changing primitives like mmap(2) (perhaps with MAP_HUGETLB ....) & munmap; you might have your own allocator and deallocator (at least for large-sized objects, or have operator new & operator delete, etc..., in some of your classes), read about C++ allocator concept. But your standard new and delete (and malloc & free for C code, often used by C++ new & delete) is using them.

Notice that most free or delete do not invoke munmap, but simply marks the released memory as reusable by future malloc or new ...

You definitely should become more familiar with garbage collection techniques and terminology. Read the GC handbook.

See also mallinfo(3) & mallopt(3) & proc(5) (perhaps use /proc/self/maps or /proc/self/smaps & /proc/self/statm from inside your program to learn about your heap, or the pmap command). Maybe strace(1) could be useful (to understand what syscalls(2) happen)

这篇关于未分配堆没有回收？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！