当 Linux 机器有更多内存时，为什么我的高内存需要代码可以在 MacOS 上运行而不是在 Linux 上? [英] Why does my high memory requiring code work on MacOS but not Linux when Linux box has more memory?
问题描述
所以我有一个程序，需要大量内存，并且需要分配(稀疏)600 000 x 600 000整数数组.尽管我认为可以在空间和内存要求方面进行改进，但是我无法在内存为40gb的计算机(Manjaro Linux x86_64)上运行该磁盘，而我的上级却可以在内存为16gb的计算机上运行该磁盘驱动器.在MacOS(64位)上.在我的计算机上运行时，程序将运行直到尝试 malloc 该数组，然后程序被杀死.
So I have a program which requires a significant amount of memory and needs to allocate a (sparse) 600 000 x 600 000 integer array. Although I believe that this can be improved with regard to space and memory requirements, I am unable to run this on my machine (Manjaro Linux x86_64) with 40gb of memory while my superior is able to run it on his machine with 16gb of memory and on MacOS (64 bit). When running on my machine the program runs until it tries to malloc that array and then the program is killed.
关于更多信息，我已经仔细检查过，这是完全相同的代码，我们正在以完全相同的方式进行编译.话虽这么说，对我来说 gcc -v 的输出是:
As for more information, I have double checked and this is exactly the same code and we are compiling it exactly the same way. That being said, the output of gcc -v for me is:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.archlinux.org/ --enable-languages=c,c++,ada,fortran,go,lto,objc,obj-c++,d --with-isl --with-linker-hash-style=gnu --with-system-zlib --enable-__cxa_atexit --enable-cet=auto --enable-checking=release --enable-clocale=gnu --enable-default-pie --enable-default-ssp --enable-gnu-indirect-function --enable-gnu-unique-object --enable-install-libiberty --enable-linker-build-id --enable-lto --enable-multilib --enable-plugin --enable-shared --enable-threads=posix --disable-libssp --disable-libstdcxx-pch --disable-libunwind-exceptions --disable-werror gdc_include_dir=/usr/include/dlang/gdc
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.2.0 (GCC)
相比之下，我上级计算机上的输出是:
By contrast, the output on my superior's computer is:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode/app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.0 (clang-1200.0.32.2)
Target: x86_64-apple-darwin19.5.0
Thread mode: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
请告知我是否还有其他相关信息.
Please let me know if there is any more relevant information I can provide.
相关的行是:
int **Idx;
Idx = (int **)malloc(Num_Idx*sizeof(*Idx));
for (i = 0; i < Num_Idx; i++)
Idx[i] = (int *)malloc(Num_Idx*sizeof(*Idx[i]));
推荐答案
Linux文档关于过量使用，重点是:
Linux内核支持以下过量使用处理模式
The Linux kernel supports the following overcommit handling modes
0-启发式过量使用处理.明显的过量使用地址空间被拒绝.用于典型的系统.它确保严重的野生分配失败，同时允许过量使用以减少交换使用量.根被允许在这种模式下分配更多的内存.这是默认值.
0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default.
1-始终过量使用.适合一些科学应用程序.经典示例是使用稀疏数组的代码并只依赖于几乎完全为零页.
1 - Always overcommit. Appropriate for some scientific applications. Classic example is code using sparse arrays and just relying on the virtual memory consisting almost entirely of zero pages.
(...)
您可以使用 sudo sysctl vm.overcommit_memory = 1
这篇关于当 Linux 机器有更多内存时，为什么我的高内存需要代码可以在 MacOS 上运行而不是在 Linux 上?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
