什么是“私人数据"?在VMMAP中定义? [英] What does "Private Data" define in VMMAP?

问题描述

我正在使用VMMap分析混合模式(托管和非托管)应用程序中的虚拟/进程地址空间利用率.我了解Windows VMM和虚拟内存API的工作原理，也了解堆内存API的工作原理.我查看了我正在使用的CRT实现(没有详细介绍)，并且(我认为我-这可能是我的败笔)了解了它如何使用上述Win32 API.

我希望了解此私人数据"统计信息向我显示的内容.我的应用程序没有直接调用任何Win32 Memory API函数，它只在本机C ++中使用"malloc/new"，在C#中使用"new"(最深处将使用Win32 Memory Management API).

VMMap提供的私有数据"的定义是:

私有内存是VirtualAlloc分配的内存，不是 由堆管理器或.NET运行时进行子分配.它 无法与其他进程共享，对系统收费 提交限制，通常包含应用程序数据.

所以我想这个定义让我问，好吧，那是谁在打电话给VirtualAlloc?是堆管理器还是.Net运行时?

我可以获取一些已提交的私有数据的地址，并使用WinDbg来查找....嗯...事实证明，微软以他们的智慧吞噬了ntdll公共符号，因此WinDbg无法正常工作很好-如果需要，我可以提供更多详细信息，但是基本上，像！address -summary这样的命令由于缺少符号而不再起作用.

表达此问题的另一种方式可能是:我可以编写哪些C ++或C#代码来导致此私有数据统计信息增加或减少?还是全部由OS，C ++运行时或.Net运行时进行管理，因此受制于异想天开?

我可以从VMMap的性质(其他内存类型互斥)中推断出此私有数据"，因此不能是以下任何一种地址空间类型:

• 堆(请注意，这包括已提交和保留的堆空间-保留的 通过对VirtualAlloc的调用，如 上面的私人数据).
• 托管堆
• 堆栈
• 可共享
• 映射文件
• 图片
• 页表
• 不可用
• 免费

(我找不到一个在线帮助文​​件，该文件定义了VMMap认为所有上述类型的东西，但是这里有一个下载帮助文件的链接:如何难道VMMap知道给定的内存区域是线程堆栈"吗?建议我可能永远找不到答案:/

更新/我发现通过打开gflags用户堆栈跟踪(gflags -i myapp.exe + ust)，可以增加私有数据的大小，我认为这是backtrace数据库，但是即使没有gflag，我仍然要努力解决私有数据.

解决方案

我知道这是一个很老的问题.但是由于某种原因，它仍然没有答案. Sasha Goldstein在DotNext会议上谈论WinDbg时提到了这个问题- https://www. youtube.com/watch?v=8t1aTbnZ2CE . 关键是可以借助WinDbg轻松地对其进行回答.

要回答CLR是否对其堆使用VirtualAlloc，我们将在此函数处设置一个断点，并使用一个脚本来打印当前堆栈(本机和托管).

bp kernelbase!VirtualAlloc ".printf \"allocating %d bytes of virtual memory\", dwo(@esp+8);.echo;  k 5; !clrstack; gc"

在这里:k 5打印本地调用堆栈的最后5帧，而!clrstack(来自SOS)打印托管堆栈. gc继续执行.

请注意，此脚本仅适用于x86进程.对于x64，您还需要其他功能(注册表和调用约定不同).

然后，我创建了一个简单的程序，该程序分配一个对象并将其添加到列表中.

    static void Main(string[] args)
    {
        var list = new List<string[]>();
        while (true) {
            var a = Console.ReadLine();
            if (a == "q" || a == "Q") break;
            var arr = new string[100];
            list.Add(arr);
        }
    }

在WinDbg下运行它并开始按Enter.在某个时候，断点命中-在List上扩展并在堆中分配额外的内存:

因此，显然CLR使用VirtualAlloc为其堆分配内存.

I am using VMMap to analyse Virtual/Process Address Space utilisation in my mixed mode (managed and unmanaged) application. I understand how the Windows VMM and the Virtual Memory API works, I understand how the Heap Memory API Works too. I have looked at the CRT implementation I am using (not in great detail) and (I think I - this could be my downfalling) understand how this uses the aforementioned Win32 APIs.

I'm looking to understand what this "Private Data" stat is showing me. My application makes no direct calls to the any of the Win32 Memory API functions, it only ever uses "malloc/new" in native C++ and "new" in C# (which deep down will be using the Win32 Memory Management API).

The definition of "Private Data" given by VMMap is:

Private memory is memory allocated by VirtualAlloc and not suballocated either by the Heap Manager or the .NET run time. It cannot be shared with other processes, is charged against the system commit limit, and typically contains application data.

So I guess this definition makes me ask, ok, so who is making the calls to VirtualAlloc? Is it the Heap Manager or .Net run time?

I could get an address of some of the committed private data and use WinDbg to find out.... Well... it turns out Microsoft in their wisdom have nobbled the ntdll public symbols, so WinDbg doesn't work so nicely - I can provide more details on this if requested, but basically commands like !address -summary no longer work due to missing symbols.

Another way to word this question might: What C++ or C# code can I write that will cause this private data statistic to increase or decrease? Or is this all managed by the OS, the C++ runtime or the .Net runtime and therefore at the mercy of it's whims?

I can deduce from the nature of VMMap (other memory types are EXCLUSIVE OF EACH OTHER) that this "private data", therefore cannot be any of the following types of address space:

• Heap (note that this includes committed AND reserved heap space - reserved through a call to VirtualAlloc, as described in the description of Private Data above).
• Managed Heap
• Stack
• Shareable
• Mapped File
• Image
• Page Table
• Unusable
• Free

(I couldn't find an online help file that defines what VMMap thinks all the above types are, but here's a link to download the help file: https://technet.microsoft.com/en-us/library/dd535533.aspx)

I have noticed that in my application, the TOTAL (reserved and committed) size of private data remains fairly constant throughout my applications lifetime, despite the Heap/Managed Heap/Stack sizes changing as expected. I have also noticed that of the ~250Mb total used by private data, only ~33Mb is actually committed. Note my method of measuring this is rather rudimentary, so the value may be changing between each of my measurements and I'm just not seeing it (if i knew what this was measuring, I could use DebugDiag to grab a dump of the process when the related counter hit a certain threshold, chicken and egg).

My current speculative theory is that this is space that is being reserved to grow the native (or managed i suppose?) heaps into as they reach their capacity, but I got nothing to prove this. So it remains firmly in the speculative pile.

Searching the internet for details on this can be painful, there are many many posts/articles/blogs out there that confuse things, use self referential definitions (the first sentence of Performance Monitor's definition for Working Set is a great example of this), are incomplete or are just plain wrong. Many places blur definitions or use an inconsistent terminology (note that VMMaps definition of the field private data, goes on to refer to it as private memory, maybe a bit of an anal complaint, but ambiguity).

Now that i've criticised the rest of the internet for getting things muddled and incorrect... if there is anything in the above that doesn't make sense or you can show me documentation to the contrary or you need a more explicit definition of, let me know and i'll put myself on the offenders list as well! I think the first half of trying to explain a memory issue to someone, online, is making sure we're all talking about the same thing.

Finally this question: How does VMMap know a given memory region is Thread Stack, specifically? suggests I may never find out an answer :/

UPDATE/EDIT: I have discovered that by turning on gflags user stack tracing (gflags -i myapp.exe +ust), you can increase the size of the Private Data, I would assume this is the backtrace database, but there even without gflags, there is still Private Data which I am struggling to account for.

解决方案

I know it's a pretty old question. But it's still unanswered for some reason. This question was mentioned by Sasha Goldstein in his talk about WinDbg at DotNext conference - https://www.youtube.com/watch?v=8t1aTbnZ2CE. The point was it can be easily answered with help of WinDbg.

To answer whether CLR uses VirtualAlloc for its heap we'll set a breakpoint at this function with a script which prints current stack (native and managed).

bp kernelbase!VirtualAlloc ".printf \"allocating %d bytes of virtual memory\", dwo(@esp+8);.echo;  k 5; !clrstack; gc"

Here: k 5 print last 5 frames of native callstack and !clrstack (from SOS) prints managed stack. gc continue execution.

Please note that this script will work only for x86 processes. For x64 you'll need some other (registries and calling convention differ).

Then I created a simple program which allocate an object and adds it to a List.

    static void Main(string[] args)
    {
        var list = new List<string[]>();
        while (true) {
            var a = Console.ReadLine();
            if (a == "q" || a == "Q") break;
            var arr = new string[100];
            list.Add(arr);
        }
    }

Run it under WinDbg and started pressing Enter. At some point the breakpoint hit - on List expanding and allocating additional memory in the heap:

So obviously CLR uses VirtualAlloc for allocating memory for its heap.

这篇关于什么是“私人数据"?在VMMAP中定义?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！