多线程程序中的意外内存泄漏 [英] Unexpected memory leak in multithread program

问题描述

我正在开发一个使用大量线程的程序，每个线程在堆中分配几兆字节的内存.当这些线程结束时，程序会保留很大一部分 RAM.

I'm working on a program using a large number of threads, each thread allocating in heap a few megabytes of memory. When these threads end, a large part of the RAM is kept by the program.

下面是一个代码示例，在 500 个线程中分配和释放 1 MB，说明了这个问题:

Here is an example of code, allocating and freeing 1 MB in 500 threads, which shows this problem:

#include <future>
#include <iostream>
#include <vector>

// filling a 1 MB array with 0
void task() {
    const size_t S = 1000000;
    int * tab = new int[S];
    std::fill(tab, tab + S, 0);
    delete[] tab;
}

int main() {
    std::vector<std::future<void>> threads;
    const size_t N = 500;

    std::this_thread::sleep_for(std::chrono::seconds(5));
    std::cout << "Starting threads" << std::endl;

    for (size_t i = 0 ; i < N ; ++i) {
        threads.push_back(std::async(std::launch::async, [=]() { return task(); }));
    }

    for (size_t i = 0 ; i < N ; ++i) {
        threads[i].get();
    }

    std::cout << "Threads ended" << std::endl;
    std::this_thread::sleep_for(std::chrono::seconds(25));

    return 0;
}

在我的计算机上，此代码仅使用 g++ -o exe main.cpp -lpthread 构建，在消息启动线程"之前使用 1976 kB，在消息线程结束"之后使用 419 MB".这些值只是示例:当我多次运行程序时，我可以获得不同的值.

On my computer, this code, simply built with g++ -o exe main.cpp -lpthread, uses 1976 kB before the message "Starting threads", and 419 MB after the message "Threads ended". These values are just examples: when I run the program multiple times, I can get different values.

我尝试过 valgrind/memcheck，但它没有检测到任何泄漏.

I have tried valgrind / memcheck, but it doesn't detect any leak.

我注意到用互斥锁锁定std::fill"操作似乎可以解决这个问题(或在很大程度上减少它)，但我不认为这是一个竞争条件问题，因为没有共享内存这里.我猜互斥锁只是在线程之间创建了一个执行顺序，以避免(或减少)内存泄漏的情况.

I have noticed that locking the "std::fill" operation with a mutex seems to solve this issue (or largely reduce it), but I don't think this is a race condition problem, as there is no shared memory here. I guess the mutex simply creates an execution order between the threads which avoid (or reduce) the conditions in which the memory leaks.

我使用的是 Ubuntu 18.04，带有 GCC 7.4.0.

I am using Ubuntu 18.04, with GCC 7.4.0.

感谢您的帮助.

奥雷利安

推荐答案

整个谜团隐藏在负责管理内存的标准库中.多线程对内存消耗有影响只是因为每个线程都需要相当多的内存(由于某些原因，大多数初学者不记得这一点).

The whole mystery is hidden in standard library which is responsible for managing memory. Mutithreading has impact on memory consumption only because each tread needs quite a lot of memory (for some reasons most beginners do not remember about that).

当您调用 delete(或 C 中的 free)时，并不意味着内存返回给系统.这只意味着标准库将这块内存标记为不再需要.

When you call delete (or free in C) it doesn't mean that memory returns to the system. It only means that standard library marks this piece of memory as not needed any more.

现在，由于向系统请求或释放内存非常昂贵，并且可以以大块的形式完成(页面大小为 8-32 kB，具体取决于硬件)，标准库尝试优化它并且不会返回所有内存立即进入系统.它假设程序可能很快再次需要这个内存.

Now since requesting or releasing memory from/to the system is quite expensive and can be done in huge chunks (page size is 8-32 kB depending on hardware), standard library tries optimize that and doesn't return all memory back to the system immediately. It assumes that program may need this memory again soon.

所以进程消耗的内存不是一个很好的数字，表明内存泄漏.只有当进程运行时间较长，保持在同一状态并不断获得内存时，才可以怀疑程序泄漏了内存.
在所有其他情况下，您应该使用 valgrind 之类的工具(我建议使用地址清理程序).

So memory consumed by process is not a good number indicating of memory leak. Only when process running for a longer time, stays in the same state and continuously gains memory, then you can suspect that program leaks memory.
In all other cases you should relay on tools like valgrind (I recommend to use address sanitizer).

还有其他优化会影响您所看到的内容.Spawning 线程的开销很大，所以当线程完成它的工作时，它并没有被完全销毁.它保存在线程池"中以备将来重用.

There is also other optimization which has impact on what you are seeing. Spawning thread is costly, so when thread completes its job, it is not destroyed completely. It is kept in a "thread pool" for future reuse.

这篇关于多线程程序中的意外内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！