为什么多个线程在持有 Mutex 时使用太多内存 [英] Why multiple threads using too much memory when holding Mutex

问题描述

下面的代码在单线程中使用了 ~150MB，但在 100 个线程中使用了几个 GB:

Below code uses ~150MB in single thread but uses several GBs in 100 threads:

use std::sync::{Arc, Mutex};
use std::thread;

fn main() {
    let f = Arc::new(Mutex::new(Foo::new("hello")));

    let mut threads = vec![];
    for i in 0..100 {
        let f = f.clone();
        let t = thread::spawn(move || loop {
            let mut locked = f.lock().unwrap();
            *locked = Foo::new("hello");
            drop(locked);
            println!("{} reloaded", i);
            thread::yield_now();
        });
        threads.push(t);
    }

    threads.into_iter().for_each(|h| h.join().unwrap());
}

pub struct Foo {
    _data: Vec<String>,
}

impl Foo {
    fn new(s: &str) -> Foo {
        Foo {
            _data: vec![s.to_owned(); 1024 * 1024],
        }
    }
}

在持有 LockGuard 时，线程应该具有独占访问权限.所以，新的 Foo 应该被分配，旧的值应该在那个时候被删除.因此，从多个线程调用时使用这么多内存对我来说没有任何意义.

While holding the LockGuard, a thread should have exclusive access. So, new Foo should be allocated and old value should be dropped at that point. So, it doesn't make any sense to me this much memory is being used when called from multiple threads.

谁能解释一下为什么这段代码要使用这么多内存?

Can anyone please explain why this code is using this much memory?

Java 中的类似代码即使有 1000 个线程也能保持大约 200mb 的内存.

Similar code in Java keeps memory ~200mb even with 1000 threads.

import java.util.ArrayList;
import java.util.List;

public class Foo {
    private List<String> data;

    public static void main(String[] args) {
        Foo f = new Foo();
        for (int i = 0; i < 1000; i++) {
            int n = i;
            new Thread(() -> {
                while (true) {
                    f.update();
                    System.gc();
                    System.out.println(n + " updated");
                }
            }).start();
        }
    }

    public synchronized void update() {
        data = new ArrayList<>(1024 * 1024);
        for (int i = 0; i < 1024 * 1024; i++) {
            data.add(new String("hello"));
        }
    }
}

推荐答案

所以问题出在大量 glibc 的 malloc 领域，每个竞技场都有预分配内存的缓存.检查它的简单方法是使用 MALLOC_ARENA_MAX=2 运行二进制文件，但最终解决方案取决于使用模式，有很多变量可以调整 glibc 的分配器:http://man7.org/linux/man-pages/man3/mallopt.3.html .

So the problem was in the big numbers of glibc's malloc arenas, every arena has cache of preallocated memory. The simple way to check it is running binary with MALLOC_ARENA_MAX=2, but final solution depend on usage pattern, there are a lot variables to tune glibc's allocator: http://man7.org/linux/man-pages/man3/mallopt.3.html .

Java 虚拟机实际上也受到 malloc 的分配器的影响.根据我的经验，有一段时间适合配置arena的数量，以防止docker内部jvm的大量内存使用.

Java virtual machine is also actually affected by malloc's allocator. From my experience some time it is suitable to configure number of arenas to prevent huge memory usage of jvm inside docker.

这篇关于为什么多个线程在持有 Mutex 时使用太多内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！