检测/诊断线程饥饿 [英] Detecting/Diagnosing Thread Starvation

问题描述

我正在对IIS应用程序进行一些性能/可伸缩性测试，有时似乎会减慢生产速度.我可以使用NUnit始终如一地重现缓慢.

I am doing some performance/scalability testing of an IIS application that occasionally seems to slow down to a crawl in production. I'm able to reproduce the slowness consistently using NUnit.

在测试过程中或生产中出现速度缓慢时，CPU和内存不会出现峰值.我强烈怀疑该应用程序正遭受线程匮乏的困扰，因为它似乎不是导致瓶颈的CPU，内存，I/O或数据库访问.我确实看到了出现 线程饥饿的迹象；例如，NLog的异步日志文件写入通常会保持长时间的静默，然后出现带有较旧时间戳记的突发事件(即，优先级较低的线程正在等待线程释放以进行写入).

CPU and Memory do not spike during the testing, or when the slowness occurs in production. My strong suspicion is that the application is suffering from thread starvation, since it does not appear to be CPU, Memory, I/O, or database access that is causing the bottleneck. I do see signs of what appear to be thread starvation; for example, NLog's async log file writes tend to have long periods of silence followed by bursts of activity with older time stamps (i.e. a lower-priority thread is waiting for threads to free up in order to write).

我可以采取哪些步骤来确定该应用程序确实是线程不足的，并且(假设是这种情况)查明导致该问题的系统的确切区域?

What steps can I take to definitively determine that the application is indeed thread starved, and (assuming that is the case) pinpoint the exact areas of the system that are causing the problem?

修改

我忽略了几乎所有代码都是同步的(这是一个遗留系统).

I neglected to mention that almost all the code is synchronous (it's a legacy system).

推荐答案

基于Sinatr的评论，我对ThreadPool.SetMinThreads和TaskCreationOptions.LongRunning进行了一些阅读，包括对

Based on Sinatr's comment, I did some reading on ThreadPool.SetMinThreads and TaskCreationOptions.LongRunning, including answers to When to use TaskCreationOptions.LongRunning?

在我的情况下，将MinThreads设置为较高的默认值会产生很大的不同.我创建了一个简单的后台进程，以查看ThreadPool中的可用线程"在测试运行过程中是否发生了显着变化并超过了MinThreads值(原来是).

Setting MinThreads to a higher default value made a huge difference in my case. I created a simple background process to see if Available Threads in the ThreadPool was changing significantly during the course of a test run and exceeding the MinThreads value (it was).

这是我用来诊断的一些代码.这不是供生产使用的，此处显示的线程使用情况的报告仅在它们最初逐渐增加时才是有趣的.另外请注意，计时器在经过时需要一个线程，因此也需要等待可用线程.

Here's some code I used to diagnose. This is not intended for production use, and the reporting of thread usage shown here would only be interesting as they ramped up initially. Also note that the Timer needs a thread when it elapses, so also needs to wait for an available thread.

静态变量:

    private static Timer _timer;
    private static int _lastActiveThreads;
    private static int _lastAvailableThreads;
    private static int _maxThreads;
    private static int _minThreads;

在启动时运行:

    int completionPortThreads;

    ThreadPool.GetMaxThreads(out _maxThreads, out completionPortThreads);
    ThreadPool.GetMinThreads(out _minThreads, out completionPortThreads);

    _timer = new Timer
    {
        AutoReset = true,
        Interval = 500,
    };

    _timer.Elapsed += TimerElasped;
    _timer.Start();

使用的方法:

    private static void TimerElasped(object sender, ElapsedEventArgs e)
    {
        int minWorkerThreads;
        int availWorkerThreads;
        int completionPortThreads;

        ThreadPool.GetMinThreads(out minWorkerThreads, out completionPortThreads);
        ThreadPool.GetAvailableThreads(out availWorkerThreads, out completionPortThreads);

        var activeThreads = _maxThreads - availWorkerThreads;

        if (availWorkerThreads != _lastAvailableThreads)
        {
            _lastAvailableThreads = availWorkerThreads;
            if (activeThreads > _lastActiveThreads)
            {
                _lastActiveThreads = activeThreads;
                Logger.Log($"+++++ Active Threads is now: {activeThreads}");

                if (activeThreads > _minThreads)
                {
                    var diff = activeThreads - _minThreads;
                    Logger.Log($"+++++ Active threads is now {activeThreads}, which is {diff} more than minThread value of {_minThreads}.  This may be causing delays.");
                }
            }
        }
    }

这篇关于检测/诊断线程饥饿的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！