如何在Docker容器内的Linux perf工具中使用调试符号? [英] How do you get debugging symbols working in linux perf tool inside Docker containers?

问题描述

我正在使用基于"ubuntu"标签的Docker容器，无法获得linux perf工具来显示调试符号.

I am using Docker containers based on the "ubuntu" tag and cannot get linux perf tool to display debugging symbols.

这就是我要演示的问题.

Here is what I'm doing to demonstrate the problem.

首先，我启动一个带有交互式外壳的容器.

First I start a container, here with an interactive shell.

$ docker run -t -i ubuntu:14.04 /bin/bash

然后从容器提示中，我安装了Linux性能工具.

Then from the container prompt I install linux perf tool.

$ apt-get update
$ apt-get install -y linux-tools-common linux-tools-generic linux-tools-`uname -r`

我现在可以使用perf工具.我的内核是3.16.0-77-generic.

I can now use the perf tool. My kernel is 3.16.0-77-generic.

现在，我将安装gcc，编译一个测试程序，然后尝试在perf record下运行它.

Now I'll install gcc, compile a test program, and try to run it under perf record.

$ apt-get install -y gcc

我将测试程序粘贴到test.c:

#include <stdio.h>

int function(int i) {
    int j;
    for(j = 2; j <= i / 2; j++) {
        if (i % j == 0) {
            return 0;
        }
    }
    return 1;
}

int main() {
    int i;
    for(i = 2; i < 100000; i++) {
        if(function(i)) {
            printf("%d\n", i);
        }
    }
}

然后编译，运行和报告:

Then compile, run, and report:

$ gcc -g -O0 test.c && perf record ./a.out && perf report

输出看起来像这样:

  72.38%  a.out  a.out              [.] 0x0000000000000544
   8.37%  a.out  a.out              [.] 0x000000000000055a
   8.30%  a.out  a.out              [.] 0x000000000000053d
   7.81%  a.out  a.out              [.] 0x0000000000000551
   0.40%  a.out  a.out              [.] 0x0000000000000540

这没有符号，即使可执行文件确实具有符号信息.

This does not have symbols, even though the executable does have symbol information.

在容器外执行相同的常规步骤可以正常工作，并显示如下内容:

Doing the same general steps outside the container works fine, and shows something like this:

96.96%  a.out  a.out             [.] function 
0.35%  a.out  libc-2.19.so       [.] _IO_file_xsputn@@GLIBC_2.2.5
0.14%  a.out  [kernel.kallsyms]  [k] update_curr
0.12%  a.out  [kernel.kallsyms]  [k] update_cfs_shares
0.11%  a.out  [kernel.kallsyms]  [k] _raw_spin_lock_irqsave                 

在主机系统中，我已经通过成为root用户并执行以下操作来打开内核符号:

In the host system I have already turned on kernel symbols by becoming root and doing:

$ echo 0 > /proc/sys/kernel/kptr_restrict 

如何使容器化版本正常工作并显示调试符号?

How do I get the containerized version to work properly and show debugging symbols?

推荐答案

使用-v /:/host标志运行容器并使用--symfs /host标志运行容器中的perf report可以解决此问题:

Running the container with -v /:/host flag and running perf report in the container with --symfs /host flag fixes it:

 96.59%  a.out  a.out              [.] function
  2.93%  a.out  [kernel.kallsyms]  [k] 0xffffffff8105144a
  0.13%  a.out  [nvidia]           [k] 0x00000000002eda57
  0.11%  a.out  libc-2.19.so       [.] vfprintf
  0.11%  a.out  libc-2.19.so       [.] 0x0000000000049980
  0.09%  a.out  a.out              [.] main
  0.02%  a.out  libc-2.19.so       [.] _IO_file_write
  0.02%  a.out  libc-2.19.so       [.] write

它不能按原样工作的部分原因? perf script的输出对此有所启发:

Part of the reason why it doesn't work as is? The output from perf script sort of sheds some light on this:

...
           a.out    24 3374818.880960: cycles:  ffffffff81141140 __perf_event__output_id_sample ([kernel.kallsyms])
           a.out    24 3374818.881012: cycles:  ffffffff817319fd _raw_spin_lock_irqsave ([kernel.kallsyms])
           a.out    24 3374818.882217: cycles:  ffffffff8109aba3 ttwu_do_activate.constprop.75 ([kernel.kallsyms])
           a.out    24 3374818.884071: cycles:            40053d [unknown] (/var/lib/docker/aufs/diff/9bd2d4389cf7ad185405245b1f5c7d24d461bd565757880bfb4f970d3f4f7915/a.out)
           a.out    24 3374818.885329: cycles:            400544 [unknown] (/var/lib/docker/aufs/diff/9bd2d4389cf7ad185405245b1f5c7d24d461bd565757880bfb4f970d3f4f7915/a.out)
...

请注意/var/lib/docker/aufs路径.它来自主机，因此它不会存在于容器中，您需要帮助perf report进行定位.之所以可能发生这种情况，是因为mmap事件是由任何cgroup外部的perf跟踪的，并且perf不会尝试重新映射路径.

Note the /var/lib/docker/aufs path. That's from the host so it won't exist in the container and you need to help perf report to locate it. This likely happens because the mmap events are tracked by perf outside of any cgroup and perf does not attempt to remap the paths.

另一个选择是在perf主机端运行，例如sudo perf record -a docker run -ti <container name>.但是集合必须在这里是系统范围的(-a标志)，因为容器是由docker daemon进程生成的，该进程不在我们在此处运行的docker客户端工具的进程层次结构中.

Another option is to run perf host-side, like sudo perf record -a docker run -ti <container name>. But the collection has to be system-wide here (the -a flag) as containers are spawned by docker daemon process which is not in the process hierarchy of the docker client tool we run here.

这篇关于如何在Docker容器内的Linux perf工具中使用调试符号?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！