由于system.currentTimeMillis()，所以系统CPU使用率很高 [英] High System CPU usage because of system.currentTimeMillis()

问题描述

我正在调试我们的风暴管理器(Wheezy计算机)上的高系统CPU使用率(而不是用户CPU使用率).这是观察结果

I was debugging high System CPU usage (Not user CPU usage) on of our storm supervisors (Wheezy machine). Here are the observations

相关过程的性能输出:

Events: 10K cpu-clock
16.40%  java  [kernel.kallsyms]   [k] system_call_after_swapgs
13.95%  java  [kernel.kallsyms]   [k] pvclock_clocksource_read
12.76%  java  [kernel.kallsyms]   [k] do_gettimeofday
12.61%  java  [vdso]              [.] 0x7ffe0fea898f
 9.02%  java  perf-17609.map      [.] 0x7fcabb8b85dc
 7.16%  java  [kernel.kallsyms]   [k] copy_user_enhanced_fast_string
 4.97%  java  [kernel.kallsyms]   [k] native_read_tsc
 2.88%  java  [kernel.kallsyms]   [k] sys_gettimeofday
 2.82%  java  libjvm.so           [.] os::javaTimeMillis()
 2.39%  java  [kernel.kallsyms]   [k] arch_local_irq_restore

在相关进程的线程中抓住了这一点

Caught this in strace of a thread of the relevant process

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.000247           0     64038           gettimeofday
  0.00    0.000000           0         1           rt_sigreturn
  0.00    0.000000           0         1           futex
------ ----------- ----------- --------- --------- ----------------
100.00    0.000247                 64040           total

最后弄清楚该线程正在while(true)中运行，并且其中一个调用是System.currentTimeMillis().我禁用了相同的功能，系统CPU％从50％下降到3％.显然，这就是问题所在.我无法理解的是，在存在vDSO的情况下，这些内核调用仅应在用户的地址空间中发生.但是，从性能报告中可以清楚地看到，内核调用确实发生在内核空间中.关于这个有什么建议吗? 内核版本:3.2.0-4-amd64 Debian 3.2.86-1 x86_64 GNU/Linux
时钟类型:kvm

Finally figured out that the thread was running in while(true) and one of the calls inside was System.currentTimeMillis(). I disabled the same and the system CPU % went down from 50% to 3%. So clearly that was the issue. What I fail to understand is, in the presence of vDSO these kernel calls should only happen in the user's address space. But as is clear from perf report, kernel calls are indeed taking place in the kernel space. Any pointers on this? Kernel version: 3.2.0-4-amd64 Debian 3.2.86-1 x86_64 GNU/Linux
clock type: kvm

添加有问题的线程的代码.

Adding code of the problematic thread.

@RequiredArgsConstructor
public class TestThread implements Runnable {
    private final Queue<String> queue;
    private final Publisher publisher;
    private final int maxBatchSize;

    private long lastPushTime;
    @Override
    public void run() {
        lastPushTime = System.currentTimeMillis();
        List<String> events = new ArrayList<>();
        while (true) {
            try {
                String message = queue.poll();
                long lastPollTime = System.currentTimeMillis();
                if (message != null) {
                    events.add(message);
                    pushEvents(events, false);
                }

                // if event threshold hasn't reached the size, but it's been there for over 10seconds, push it.
                if ((lastPollTime - lastPushTime > 10000) && (events.size() > 0)) {
                    pushEvents(events, true);
                }
            } catch (Exception e) {
                // Log and do something
            }
        }
    }

    private void pushEvents(List<String> events, boolean forcePush) {
        if (events.size() >= maxBatchSize || forcePush) {
            pushToHTTPEndPoint(events);
            events.clear();
            lastPushTime = System.currentTimeMillis();
        }
    }

    private void pushToHTTPEndPoint(List<String> events) {
        publisher.publish(events);
    }
}

推荐答案

我无法理解的是，在存在vDSO的情况下，这些内核调用应仅在用户的地址空间中发生.但是，从性能报告中可以明显看出，内核调用确实发生在内核空间中.有任何指针吗?

What I fail to understand is, in the presence of vDSO these kernel calls should only happen in the user's address space. But as is clear from perf report, kernel calls are indeed taking place in the kernel space. Any pointers on this?

可以在虚拟系统上禁用

vDSO. KVM使用PVClock(您可以在此精美的文章)，它取决于内核版本. 例如，我们可以在此处 VCLOCK_MODE永远不会被覆盖. 另一方面，请此处更改了vclock_mode-和vclock_mode 指标也适用于vDSO.

vDSO could be disabled on a virtual system. KVM uses PVClock (you could read more about in this nice article) and it depends on kernel version. For example, we could see here that VCLOCK_MODE is never overridden. On the other hand, here it is changed vclock_mode - and vclock_mode indicator for vDSO too.

此支持是在 commit 中引入的，并已在3.8版本中发布

This support was introduced in this commit and released in 3.8 version of Linux kernel.

通常，在我的实践中，如果长时间在"while(true)"内部调用某些内容，则总是会看到大量的CPU消耗.

Generally, in my practice, if you call something inside "while(true)" for a long time, you will always see a big CPU consumption.

当然，在大多数情况下，阻塞队列就足够了，但是如果需要良好的延迟和性能，也可以在没有线程阻塞的情况下使用自旋，但是您应该限制旋转周期并制定基准来衡量此优化的影响.元代码可能类似于:

Of course, Blocking Queue is enough in most cases, but if you need good latency and performance, you could use spinning too, without thread blocking, but you should limit spin cycles and make benchmarks to measure the impact of this optimization. The meta code could be something like:

int spin = 100;
while(spin-- > 0) {
    // try to get result
}
// still no result -> execute blocking code

这篇关于由于system.currentTimeMillis()，所以系统CPU使用率很高的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！