C ++:在程序，Windows和Linux中测量内存使用情况 [英] C++: Measuring memory usage from within the program, Windows and Linux

问题描述

看，我想测量我的C ++程序的内存使用情况.从程序内部，没有探查器或进程查看器等.

See, I wanted to measure memory usage of my C++ program. From inside the program, without profilers or process viewers, etc.

为什么从程序内部?

1. 测量将完成数千次-必须自动化.因此，只要注意Task Manager，top等，就不会这样做
2. 要在生产运行期间进行测量 –由于运行时间已经可以忽略不计(对于大型问题实例，需要几个小时)，所以由分析器导致的性能下降是不可接受的. /li>

1. Measurements will be done thousands of times—must be automated; therefore, having an eye on Task Manager, top, whatever, will not do
2. Measurements are to be done during production runs—performance degradation, which may be caused by profilers, is not acceptable since the run times are non-negligible already (several hours for large problem instances)

注释.为什么要进行测量?相对于预先计算预期"使用情况，测量使用的内存(由操作系统报告)的唯一原因是我无法直接分析地"sizeof"多少我的主要数据结构使用的是什么.结构本身是

note. Why measure at all? The only reason to measure used memory (as reported by the OS) as opposed to calculating "expected" usage in advance is the fact that I can not directly, analytically "sizeof" how much does my principal data structure use. The structure itself is

unordered_map<bitset, map<uint16_t, int64_t> >

被打包到vector中(list实际上也足够，我只需要访问邻居"结构；如果没有有关内存使用的详细信息，我几乎无法决定要使用哪一个)选择)

these are packed into a vector for all I care (a list would actually suffice as well, I only ever need to access the "neighbouring" structures; without details on memory usage, I can hardly decide which to choose)

vector< unordered_map<bitset, map<uint16_t, int64_t> > >

因此，如果有人知道如何调整"这种结构所占用的内存，那也将解决问题(尽管我可能不得不提出这个问题或其他东西).

so if anybody knows how to "sizeof" the memory occupied by such a structure, that would also solve the issue (though I'd probably have to fork the question or something).

环境::可以假定该程序全部在给定的机器上运行(当然，连同OS等；也可以是PC或超级计算机的节点)；它肯定是唯一需要大量内存(例如，> 512 MiB)的程序-计算实验环境.该程序可以在我的家用PC(16GiB RAM； Windows 7或Linux Mint 18.1)或机构超级计算机的节点(大约100GiB RAM，CentOS 7)上运行，程序可能希望消耗掉所有这些内存. 请注意，超级计算机有效地禁止了用户进程的磁盘交换，并且我的家用PC的页面文件很小.

Environment: It may be assumed that the program runs all alone on the given machine (along with the OS, etc. of course; either a PC or a supercomputer's node); it is certain to be the only one program requiring large (say > 512 MiB) amounts of memory—computational experiment environment. The program is either run on my home PC (16GiB RAM; Windows 7 or Linux Mint 18.1) or the institution supercomputer's node (circa 100GiB RAM, CentOS 7), and the program may want to consume all that RAM. Note that the supercomputer effectively prohibits disk swapping of user processes, and my home PC has a smallish page file.

内存使用模式.可以说该程序顺序填充一种表格，每行其中是上述的vector<...>.假设主要数据结构称为supp.然后，对于每个整数k，要填充 supp[k]，都需要supp[k-1]中的数据.填充supp[k]时，它用于初始化 supp[k+1].因此，每次都必须易于访问 this ，上一个和下一个表行".填满表格后，程序会在表格中进行相对较快的(与初始化"并填满表格相比)非穷举搜索，从而获得解决方案. 请注意，该内存仅通过STL容器分配，我自己从来没有明确地new()或malloc().

Memory usage pattern. The program can be said to sequentially fill a sort of table, each row wherein is the vector<...> as specified above. Say the prime data structure is called supp. Then, for each integer k, to fill supp[k], the data from supp[k-1] is required. As supp[k] is filled it is used to initialize supp[k+1]. Thus, at each time, this, prev, and next "table rows" must be readily accessible. After the table is filled, the program does a relatively quick (compared with "initializing" and filling the table), non-exhaustive search in the table, through which a solution is obtained. Note that the memory is only allocated through the STL containers, I never explicitly new() or malloc() myself.

问题.一厢情愿.

1. 从其源代码内部测量进程的总内存使用量(包括交换到磁盘)的适当方法是什么(一种用于Windows，一种用于Linux) ?
2. 应该是另一个问题，或者是一个很好的谷歌搜索会话，但是仍然-明确控制(例如鼓励或劝阻)交换到磁盘的正确(或简单)方法是什么?我们欢迎您提供有关该主题的权威书籍的指针.再次，请原谅我的无知，我想用一种方式在永不交换supp"或 交换supp[10]"；然后，在需要时，"unswap supp[10]"(全部交换)从程序的代码中进行.我以为我必须决定序列化数据结构并将其显式存储为二进制文件，然后逆转转换.

1. What is the appropriate way to measure total memory usage (including swapped to disk) of a process from inside its source code (one for Windows, one for Linux)?
2. Should probably be another question, or rather a good googling session, but still---what is the proper (or just easy) way to explicitly control (say encourage or discourage) swapping to disk? A pointer to an authoritative book on the subject would be very welcome. Again, forgive my ignorance, I'd like a means to say something on the lines of "NEVER swap supp" or "swap supp[10]"; then, when I need it, "unswap supp[10]"—all from the program's code. I thought I'd have to resolve to serialize the data structures and explicitly store them as a binary file, then reverse the transformation.

---

在Linux上，似乎最简单的方法是仅通过sbrk(0)捕获堆指针，将其转换为64位无符号整数，并在分配内存后计算差值，这种方法产生了合理的结果(不是做更严格的测试).

---

On Linux, it appeared the easiest to just catch the heap pointers through sbrk(0), cast them as 64-bit unsigned integers, and compute the difference after the memory gets allocated, and this approach produced plausible results (did not do more rigorous tests yet).

编辑5..删除了对HeapAlloc争吵的引用-不相关.

edit 5. Removed reference to HeapAlloc wrangling—irrelevant.

编辑4. Windows解决方案 此段代码报告了与任务管理器中的工作集匹配的工作集；这就是我想要的全部内容-在Windows 10 x64上进行了测试(通过new uint8_t[1024*1024]或new uint8_t[1ULL << howMuch]之类的分配进行了测试，还没有在我的产品"中进行测试). 在Linux上，我会尝试getrusage或类似的方法. 正如@IInspectable和@conio

edit 4. Windows solution This bit of code reports the working set that matches the one in Task Manager; that's about all I wanted—tested on Windows 10 x64 (tested by allocations like new uint8_t[1024*1024], or rather, new uint8_t[1ULL << howMuch], not in my "production" yet ). On Linux, I'd try getrusage or something to get the equivalent. The principal element is GetProcessMemoryInfo, as suggested by @IInspectable and @conio

#include<Windows.h>
#include<Psapi.h>
//get the handle to this process
auto myHandle = GetCurrentProcess();
//to fill in the process' memory usage details
PROCESS_MEMORY_COUNTERS pmc;
//return the usage (bytes), if I may
if (GetProcessMemoryInfo(myHandle, &pmc, sizeof(pmc)))
    return(pmc.WorkingSetSize);
else
    return 0;

编辑5.删除了对GetProcessWorkingSetSize的不相关引用.谢谢@conio.

edit 5. Removed reference to GetProcessWorkingSetSize as irrelevant. Thanks @conio.

推荐答案

要了解您的流程占用了多少物理内存，您需要查询流程工作集或更可能是私有工作集.工作集是(或多或少)您的进程使用的RAM中的物理页面数量.私有工作集不包括共享内存.

To know how much physical memory your process takes you need to query the process working set or, more likely, the private working set. The working set is (more or less) the amount of physical pages in RAM your process uses. Private working set excludes shared memory.

请参见

• 什么是专用字节，虚拟字节，工作集?
• 如何解释Windows任务管理器?

• What is private bytes, virtual bytes, working set?
• How to interpret Windows Task Manager?
• https://blogs.msdn.microsoft.com/tims/2010/10/29/pdc10-mysteries-of-windows-memory-management-revealed-part-two/

了解术语和更多细节.

两个指标都有性能计数器.

There are performance counters for both metrics.

(您也可以使用QueryWorkingSet(Ex)并自行计算，但是我认为这很讨厌.您可以使用GetProcessMemoryInfo获得(非私有)工作集.)

(You can also use QueryWorkingSet(Ex) and calculate that on your own, but that's just nasty in my opinion. You can get the (non-private) working set with GetProcessMemoryInfo.)

但是更有趣的问题是，这是否有助于您的程序做出有用的决定.如果没有人要求或使用内存，那么仅使用大多数物理内存这一事实就不会引起人们的兴趣.还是您只是担心程序占用太多内存?

But the more interesting question is whether or not this helps your program to make useful decisions. If nobody's asking for memory or using it, the mere fact that you're using most of the physical memory is of no interest. Or are you worried about your program alone using too much memory?

关于它采用的算法或其内存使用模式，您什么也没说.如果它使用大量内存，但是大多数情况下是顺序执行的，并且相对很少地返回到旧内存，这可能不是问题. Windows急切地将旧"页面写入磁盘，然后完全调出常驻页面才能满足对物理内存的需求.如果一切顺利，将已写入磁盘页面中的这些资源重用于其他用途确实很便宜.

You haven't said anything about the algorithms it employs or its memory usage patterns. If it uses lots of memory, but does this mostly sequentially, and comes back to old memory relatively rarely it might not be a problem. Windows writes "old" pages to disk eagerly, before paging out resident pages is completely necessary to supply demand for physical memory. If everything goes well, reusing these already written to disk pages for something else is really cheap.

如果您真正关心的是内存抖动(由于交换开销，虚拟内存将无用")，那么这就是您应该寻找的内容，而不是试图从中推断(或猜测...)使用的物理内存量.一个更有用的度量标准是每单位时间的页面错误.碰巧也有针对此的性能计数器.例如，请参见评估内存和缓存使用情况.

If your real concern is memory thrashing ("virtual memory will be of no use due to swapping overhead"), then this is what you should be looking for, rather than trying to infer (or guess...) that from the amount of physical memory used. A more useful metric would be page faults per unit of time. It just so happens that there are performance counters for this too. See, for example Evaluating Memory and Cache Usage.

我怀疑这是您做出决定的更好指标.

I suspect this to be a better metric to base your decision on.

这篇关于C ++:在程序，Windows和Linux中测量内存使用情况的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！