Linux:由于虚拟内存限制，在一个进程中不能分配超过32 GB/64 GB的内存 [英] Linux: Cannot allocate more than 32 GB/64 GB of memory in a single process due to virtual memory limit

问题描述

我有一台具有128 GB RAM的计算机，运行Linux(3.19.5-200.fc21.x86_64).但是，我不能在单个进程中分配大约30 GB以上的RAM.除此之外，malloc失败:

I have a computer with 128 GB of RAM, running Linux (3.19.5-200.fc21.x86_64). However, I cannot allocate more than ~30 GB of RAM in a single process. Beyond this, malloc fails:

#include <stdlib.h>
#include <iostream>

int main()
{
   size_t gb_in_bytes = size_t(1)<<size_t(30); // 1 GB in bytes (2^30).
   // try to allocate 1 block of 'i' GB.
   for (size_t i = 25; i < 35; ++ i) {
      size_t n = i * gb_in_bytes;
      void *p = ::malloc(n);
      std::cout << "allocation of 1 x " << (n/double(gb_in_bytes)) << " GB of data. Ok? " << ((p==0)? "nope" : "yes") << std::endl;
      ::free(p);
   }
}

这将产生以下输出:

/tmp> c++ mem_alloc.cpp && a.out

allocation of 1 x 25 GB of data. Ok? yes
allocation of 1 x 26 GB of data. Ok? yes
allocation of 1 x 27 GB of data. Ok? yes
allocation of 1 x 28 GB of data. Ok? yes
allocation of 1 x 29 GB of data. Ok? yes
allocation of 1 x 30 GB of data. Ok? yes
allocation of 1 x 31 GB of data. Ok? nope
allocation of 1 x 32 GB of data. Ok? nope
allocation of 1 x 33 GB of data. Ok? nope
allocation of 1 x 34 GB of data. Ok? nope

我搜索了很长时间，发现这与最大虚拟内存大小有关:

I searched for quite some time, and found that this is related to the maximum virtual memory size:

~> ulimit -all
[...]
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
virtual memory          (kbytes, -v) 32505856
[...]

我可以通过ulimit -v 64000000将此限制增加到〜64 GB，但是不能再增加了.除此之外，我还会收到operation not permitted错误:

I can increase this limit to ~64 GB via ulimit -v 64000000, but not further. Beyond this, I get operation not permitted errors:

~> ulimit -v 64000000
~> ulimit -v 65000000                                                                                                                                  
bash: ulimit: virtual memory: cannot modify limit: Operation not permitted                                                                              
~> ulimit -v unlimited
bash: ulimit: virtual memory: cannot modify limit: Operation not permitted 

更多搜索显示，原则上应该可以通过/etc/security/limits.conf中的"as"(地址空间)条目设置这些限制.但是，通过这样做，我只能减少最大虚拟内存量，而不能增加它.

Some more searching revealed that in principle it should be possible to set these limits via the "as" (address space) entry in /etc/security/limits.conf. However, by doing this, I could only reduce the maximum amount of virtual memory, not increase it.

是否有办法完全取消每个进程的虚拟内存限制，或将其增加到超过64 GB?我想在一个应用程序中使用所有物理内存.

Is there any way to either lift this limit of virtual memory per process completely, or to increase it beyond 64 GB? I would like to use all of the physical memory in a single application.

• 在Ingo Leonhardt之后，我以root用户而不是标准用户身份登录后尝试了ulimits -v unlimited.这样做可以解决root用户的问题(然后，该程序可以在以root用户身份登录时分配所有物理内存).但这仅适用于root用户，不适用于其他用户.但是，至少这意味着原则上内核可以很好地处理此问题，并且仅存在配置问题.

• Following Ingo Leonhardt, I tried ulimits -v unlimited after logging in as root, not as standard user. Doing this solves the problem for root (the program can then allocate all the physical memory while logged in as root). But this works only for root, not for other users. However, at the least this means that in principle the kernel can handle this just fine, and that there is only a configuration problem.

关于limits.conf:我尝试显式添加

• 坚如磐石
• 软如无限

到/etc/security/limits.conf，然后重新启动.这没有效果.以标准用户身份登录后，ulimit -v仍返回约32 GB，并且ulimit -v 65000000仍显示permission denied(ulimit -v 64000000起作用). limits.conf的其余部分已被注释掉，在/etc/security/limits.d中只有一个其他不相关的条目(对于非root用户，将nproc限制为4096).也就是说，虚拟内存限制必须来自limits.conf以外的其他位置.还有什么想法可能导致ulimits -v不受限制?

to /etc/security/limits.conf, and rebooting. This had no effect. After login as standard user, ulimit -v still returns about 32 GB, and ulimit -v 65000000 still says permission denied (while ulimit -v 64000000 works). The rest of limits.conf is commented out, and in /etc/security/limits.d there is only one other, unrelated entry (limiting nproc to 4096 for non-root users). That is, the virtual memory limit must be coming from elsewhere than limits.conf. Any ideas what else could lead to ulimits -v not being "unlimited"?

编辑/分辨率:

• 这是我自己的愚蠢造成的.我的用户设置中有一个(长期被遗忘的)程序，该程序使用setrlimit来限制每个进程的内存量，以防止Linux转换为死亡.它被无意间从32 GB的计算机复制到128 GB的计算机.感谢Paul和Andrew Janke以及其他所有人的帮助来进行跟踪.对不起大家:/.

• It was caused by my own stupidity. I had a (long forgotten) program in my user setup which used setrlimit to restrict the amount of memory per process to prevent Linux from swapping to death. It was unintentionally copied from a 32 GB machine to the 128 GB machine. Thanks to Paul and Andrew Janke and everyone else for helping to track it down. Sorry everyone :/.

如果其他任何人遇到此问题:在bash和配置文件设置中搜索ulimit/setrlimit，并搜索可能调用它们的程序(您自己的设置和系统范围的/etc设置)，并确保/security/limits.conf执行不包括此限制...(或至少尝试创建一个新用户，以查看是否在您的用户或系统设置程序中发生了此情况)

If anyone else encounters this: Search for ulimit/setrlimit in the bash and profile settings, and programs potentially calling those (both your own and the system-wide /etc settings) and make sure that /security/limits.conf does not include this limit... (or at least try creating a new user, to see if this happens in your user or the system setup)

推荐答案

这是一个ulimit和系统设置问题，而不是c ++问题.

This is a ulimit and system setup problem, not a c++ problem.

我可以毫无问题地在Amazon EC2实例类型r3.4xlarge上运行经过适当修改的代码.在现货市场上，它们的价格不到每小时0.20美元，因此我建议您租用一个，然后在/etc中浏览一下并与您自己的设置进行比较...或者您可能需要重新编译Linux内核才能使用它大量的内存...但这不是C ++或gcc问题.

I can run your appropriately modified code on an Amazon EC2 instance type r3.4xlarge with no problem. These cost less than $0.20/hour on the spot market, and so I suggest you rent one, and perhaps take a look around in /etc and compare to your own setup... or maybe you need to recompile a Linux kernel to use that much memory... but it is not a C++ or gcc problem.

EC2计算机上的Ubuntu已设置为无限制的进程内存.

Ubuntu on the EC2 machine was already set up for unlimited process memory.

$ sudo su
# ulimit -u
--> unlimited

这个有125GB的内存

This one has 125GB of ram

# free
             total       used       free     shared    buffers     cached
Mem:     125903992    1371828  124532164        344      22156     502248
-/+ buffers/cache:     847424  125056568
Swap:            0          0          0

我已将您程序的限制修改为最大149GB.

I modified the limits on your program to go up to 149GB.

这是输出.看起来不错，高达118GB.

Here's the output. Looks good up to 118GB.

root@ip-10-203-193-204:/home/ubuntu# ./memtest
allocation of 1 x 25 GB of data. Ok? yes
allocation of 1 x 26 GB of data. Ok? yes
allocation of 1 x 27 GB of data. Ok? yes
allocation of 1 x 28 GB of data. Ok? yes
allocation of 1 x 29 GB of data. Ok? yes
allocation of 1 x 30 GB of data. Ok? yes
allocation of 1 x 31 GB of data. Ok? yes
allocation of 1 x 32 GB of data. Ok? yes
allocation of 1 x 33 GB of data. Ok? yes
allocation of 1 x 34 GB of data. Ok? yes
allocation of 1 x 35 GB of data. Ok? yes
allocation of 1 x 36 GB of data. Ok? yes
allocation of 1 x 37 GB of data. Ok? yes
allocation of 1 x 38 GB of data. Ok? yes
allocation of 1 x 39 GB of data. Ok? yes
allocation of 1 x 40 GB of data. Ok? yes
allocation of 1 x 41 GB of data. Ok? yes
allocation of 1 x 42 GB of data. Ok? yes
allocation of 1 x 43 GB of data. Ok? yes
allocation of 1 x 44 GB of data. Ok? yes
allocation of 1 x 45 GB of data. Ok? yes
allocation of 1 x 46 GB of data. Ok? yes
allocation of 1 x 47 GB of data. Ok? yes
allocation of 1 x 48 GB of data. Ok? yes
allocation of 1 x 49 GB of data. Ok? yes
allocation of 1 x 50 GB of data. Ok? yes
allocation of 1 x 51 GB of data. Ok? yes
allocation of 1 x 52 GB of data. Ok? yes
allocation of 1 x 53 GB of data. Ok? yes
allocation of 1 x 54 GB of data. Ok? yes
allocation of 1 x 55 GB of data. Ok? yes
allocation of 1 x 56 GB of data. Ok? yes
allocation of 1 x 57 GB of data. Ok? yes
allocation of 1 x 58 GB of data. Ok? yes
allocation of 1 x 59 GB of data. Ok? yes
allocation of 1 x 60 GB of data. Ok? yes
allocation of 1 x 61 GB of data. Ok? yes
allocation of 1 x 62 GB of data. Ok? yes
allocation of 1 x 63 GB of data. Ok? yes
allocation of 1 x 64 GB of data. Ok? yes
allocation of 1 x 65 GB of data. Ok? yes
allocation of 1 x 66 GB of data. Ok? yes
allocation of 1 x 67 GB of data. Ok? yes
allocation of 1 x 68 GB of data. Ok? yes
allocation of 1 x 69 GB of data. Ok? yes
allocation of 1 x 70 GB of data. Ok? yes
allocation of 1 x 71 GB of data. Ok? yes
allocation of 1 x 72 GB of data. Ok? yes
allocation of 1 x 73 GB of data. Ok? yes
allocation of 1 x 74 GB of data. Ok? yes
allocation of 1 x 75 GB of data. Ok? yes
allocation of 1 x 76 GB of data. Ok? yes
allocation of 1 x 77 GB of data. Ok? yes
allocation of 1 x 78 GB of data. Ok? yes
allocation of 1 x 79 GB of data. Ok? yes
allocation of 1 x 80 GB of data. Ok? yes
allocation of 1 x 81 GB of data. Ok? yes
allocation of 1 x 82 GB of data. Ok? yes
allocation of 1 x 83 GB of data. Ok? yes
allocation of 1 x 84 GB of data. Ok? yes
allocation of 1 x 85 GB of data. Ok? yes
allocation of 1 x 86 GB of data. Ok? yes
allocation of 1 x 87 GB of data. Ok? yes
allocation of 1 x 88 GB of data. Ok? yes
allocation of 1 x 89 GB of data. Ok? yes
allocation of 1 x 90 GB of data. Ok? yes
allocation of 1 x 91 GB of data. Ok? yes
allocation of 1 x 92 GB of data. Ok? yes
allocation of 1 x 93 GB of data. Ok? yes
allocation of 1 x 94 GB of data. Ok? yes
allocation of 1 x 95 GB of data. Ok? yes
allocation of 1 x 96 GB of data. Ok? yes
allocation of 1 x 97 GB of data. Ok? yes
allocation of 1 x 98 GB of data. Ok? yes
allocation of 1 x 99 GB of data. Ok? yes
allocation of 1 x 100 GB of data. Ok? yes
allocation of 1 x 101 GB of data. Ok? yes
allocation of 1 x 102 GB of data. Ok? yes
allocation of 1 x 103 GB of data. Ok? yes
allocation of 1 x 104 GB of data. Ok? yes
allocation of 1 x 105 GB of data. Ok? yes
allocation of 1 x 106 GB of data. Ok? yes
allocation of 1 x 107 GB of data. Ok? yes
allocation of 1 x 108 GB of data. Ok? yes
allocation of 1 x 109 GB of data. Ok? yes
allocation of 1 x 110 GB of data. Ok? yes
allocation of 1 x 111 GB of data. Ok? yes
allocation of 1 x 112 GB of data. Ok? yes
allocation of 1 x 113 GB of data. Ok? yes
allocation of 1 x 114 GB of data. Ok? yes
allocation of 1 x 115 GB of data. Ok? yes
allocation of 1 x 116 GB of data. Ok? yes
allocation of 1 x 117 GB of data. Ok? yes
allocation of 1 x 118 GB of data. Ok? yes
allocation of 1 x 119 GB of data. Ok? nope
allocation of 1 x 120 GB of data. Ok? nope
allocation of 1 x 121 GB of data. Ok? nope
allocation of 1 x 122 GB of data. Ok? nope
allocation of 1 x 123 GB of data. Ok? nope
allocation of 1 x 124 GB of data. Ok? nope
allocation of 1 x 125 GB of data. Ok? nope
allocation of 1 x 126 GB of data. Ok? nope
allocation of 1 x 127 GB of data. Ok? nope
allocation of 1 x 128 GB of data. Ok? nope
allocation of 1 x 129 GB of data. Ok? nope
allocation of 1 x 130 GB of data. Ok? nope
allocation of 1 x 131 GB of data. Ok? nope
allocation of 1 x 132 GB of data. Ok? nope
allocation of 1 x 133 GB of data. Ok? nope
allocation of 1 x 134 GB of data. Ok? nope
allocation of 1 x 135 GB of data. Ok? nope
allocation of 1 x 136 GB of data. Ok? nope
allocation of 1 x 137 GB of data. Ok? nope
allocation of 1 x 138 GB of data. Ok? nope
allocation of 1 x 139 GB of data. Ok? nope
allocation of 1 x 140 GB of data. Ok? nope
allocation of 1 x 141 GB of data. Ok? nope
allocation of 1 x 142 GB of data. Ok? nope
allocation of 1 x 143 GB of data. Ok? nope
allocation of 1 x 144 GB of data. Ok? nope
allocation of 1 x 145 GB of data. Ok? nope
allocation of 1 x 146 GB of data. Ok? nope
allocation of 1 x 147 GB of data. Ok? nope
allocation of 1 x 148 GB of data. Ok? nope
allocation of 1 x 149 GB of data. Ok? nope

现在，我在这上面花了约0.17美元...

Now, about that US$0.17 I spent on this...

这篇关于Linux:由于虚拟内存限制，在一个进程中不能分配超过32 GB/64 GB的内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！