malloc 在 C 中可以有多大? [英] How big can a malloc be in C?

问题描述

我在 C 中有一个 malloc，它是 26901^2*sizeof(double)

I have a malloc in C that is 26901^2*sizeof(double)

这让我想到这里最大的价值是多少?

This got me thinking what the largest value can be here?

另外，我在定义一个宏来访问这个二维数组时会遇到什么问题吗?

Also, would I have any problems defining a macro to access this 2D array?

 #define DN(i,j) ((int)i * ny + (int)j)

因为这似乎对我不起作用 - 或者我至少不确定它是否有效.我不知道如何在宏上进行全视图潜水以告诉我 A[DN(indx,jndx)] 实际在看什么.

Because this seems to not be working for me - or I am at least unsure it is. I can't figure out how to make totalview dive on a macro to tell me what A[DN(indx,jndx)] is actually looking at.

推荐答案

观察

假设一个典型的分配器，例如 glibc 使用的那个，有一些观察结果:

Observations

Assuming a typical allocator, such as the one glibc uses, there are some observations:

1. 无论内存是否实际使用，该区域都必须在虚拟内存中连续保留.
2. 最大的空闲连续区域取决于现有内存区域的内存使用情况，以及这些区域对 malloc.
3. 映射做法取决于架构和操作系统.此外，获取内存区域的底层系统调用会受到这些做法的影响(例如 malloc 调用到 mmap 获取页面).

实验

这是一个 简单程序来分配可能的最大块(使用 gcc large_malloc_size.c -Wall -O2 编译:

Experiment

Here's a simple program to allocate the largest possible block (compile with gcc largest_malloc_size.c -Wall -O2:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

static void *malloc_wrap(size_t size)
{
    void *p = malloc(size);
    if (p) {
        printf("Allocated %zu bytes from %p to %p
", size, p, p + size);
    }
    else {
        printf("Failed to allocated %zu bytes
", size);
    }
    return p;
}

int main()
{
    size_t step = 0x1000000;
    size_t size = step;
    size_t best = 0;
    while (step > 0)
    {
        void *p = malloc_wrap(size);
        if (p) {
            free(p);
            best = size;
        }
        else {
            step /= 0x10;
        }
        size += step;
    }
    void *p = malloc_wrap(best);
    if (p) {
        pause();
        return 0;
    }
    else {
        return 1;
    }
}

在我的 Linux stanley 2.6.32-24-generic-pae #39-Ubuntu SMP 上运行上述程序 (./a.out) 7 月 28 日星期三 07:39:26UTC 2010 i686 GNU/Linux 机器得到这个结果:

Running the above program (./a.out) on my Linux stanley 2.6.32-24-generic-pae #39-Ubuntu SMP Wed Jul 28 07:39:26 UTC 2010 i686 GNU/Linux machine obtains this result:

<snip>
Allocated 2919235584 bytes from 0x9763008 to 0xb7763008
Allocated 2936012800 bytes from 0x8763008 to 0xb7763008
Failed to allocated 2952790016 bytes
Failed to allocated 2953838592 bytes
Failed to allocated 2953904128 bytes
Failed to allocated 2953908224 bytes
Allocated 2936012800 bytes from 0x85ff008 to 0xb75ff008

这是正好 2800MiB 的分配.从 /proc/[number]/maps:

This is an allocation of exactly 2800MiB. Observing the relevant mapping from /proc/[number]/maps:

<snip>
0804a000-0804b000 rw-p 00001000 08:07 3413394    /home/matt/anacrolix/public/stackoverflow/a.out
085ff000-b7600000 rw-p 00000000 00:00 0          [heap]
b7600000-b7621000 rw-p 00000000 00:00 0 
b7621000-b7700000 ---p 00000000 00:00 0 
b7764000-b7765000 rw-p 00000000 00:00 0 
b7765000-b78b8000 r-xp 00000000 08:08 916041     /lib/tls/i686/cmov/libc-2.11.1.so
<snip>
bfc07000-bfc1c000 rw-p 00000000 00:00 0          [stack]

结论

看起来堆在程序数据和代码之间的区域已经扩展，共享库映射与 用户/内核内存空间边界(在这个系统上显然是 3G/1G).

Conclusion

It appears the heap has been expanded in the area between the program data and code, and the shared library mappings, which sit snug against the user/kernel memory space boundary (obviously 3G/1G on this system).

这个结果表明使用malloc的最大可分配空间大致等于:

This result suggests that the maximum allocatable space using malloc is roughly equal to:

1. 用户空间区域(示例中为 3GB)
2. 减去到堆开始的偏移量(程序代码和数据)
3. 为主线程堆栈保留的空间更少
4. 减少共享库中所有映射占用的空间
5. 最后，堆可用区域内的底层系统调用可以找到的最大连续区域(可能被其他映射分段)

注意事项

关于 glibc 和 Linux 实现，以下手动片段非常有趣:

Notes

With respect to glibc and Linux implementations, the following manual snippets are of great interest:

malloc

   Normally, malloc() allocates memory from the heap, and adjusts the size
   of the heap as required, using sbrk(2).  When allocating blocks of mem‐
   ory larger than MMAP_THRESHOLD bytes, the glibc malloc() implementation
   allocates the memory as a  private  anonymous  mapping  using  mmap(2).
   MMAP_THRESHOLD  is  128  kB  by  default,  but is adjustable using mal‐
   lopt(3).

mmap

   MAP_ANONYMOUS
          The mapping is not backed by any file; its contents are initial‐
          ized to zero.

后记

此测试是在 x86 内核上完成的.我希望 x86_64 内核会产生类似的结果，尽管返回的内存区域要大得多.其他操作系统的映射位置和大型 malloc 的处理可能会有所不同，因此结果可能会有很大差异.

Afterword

This test was done on a x86 kernel. I'd expect similar results from a x86_64 kernel, albeit with vastly larger memory regions returned. Other operating systems may vary in their placement of mappings, and the handling of large mallocs, so results could be quite considerably different.

这篇关于malloc 在 C 中可以有多大?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！