为什么使用任务集在一组隔离的内核上运行多线程Linux程序会导致所有线程在一个内核上运行? [英] Why does using taskset to run a multi-threaded Linux program on a set of isolated cores cause all threads to run on one core?

问题描述

所需的行为:在一组使用 isolcpus 隔离的内核上运行多线程Linux程序.

这是一个小程序，我们可以将其用作示例多线程程序:

  #include< stdio.h>#include< pthread.h>#include< err.h>#include< unistd.h>#include< stdlib.h>#定义NTHR 16#定义时间60 * 5空白 *do_stuff(void * arg){int i = 0;(void)arg;而(1){我+ =我;usleep(10000);/*不要控制CPU */}}整型主(无效){pthread_t线程[NTHR];int rv，i;对于(i = 0; i< NTHR; i ++){rv = pthread_create(& threads [i]，NULL，do_stuff，NULL);如果(rv){perror("pthread_create");返回(EXIT_FAILURE);}}睡觉时间);退出(EXIT_SUCCESS);} 

如果我在没有隔离CPU的内核上编译并运行它，那么线程将分散到我的4个CPU上.好！

现在，如果我将 isolcpus = 2,3 添加到内核命令行并重新启动:

• 在没有任务集的情况下运行程序会在内核0和1上分配线程.这是可以预期的，因为默认的相似性掩码现在不包括内核2和3.
• 使用 taskset -c 0,1 运行具有相同的效果.好.
• 使用 taskset -c 2,3 运行会导致所有线程进入同一个内核(内核2或3).这是不希望的.线程应该分布在核心2和3上.对吗?

此帖子描述了类似的内容问题(尽管给出的示例距离pthreads API更远).OP乐于通过使用其他调度程序来解决此问题.我不确定这对我的用例是否理想.

是否有一种方法可以使用默认调度程序将线程分布在隔离的内核上?

这是我应该报告的内核错误吗?

编辑:

如果您使用像fifo调度程序这样的实时调度程序，确实会发生正确的事情.有关详细信息，请参见 man sched 和 man chrt .

解决方案

来自Linux内核参数文档:

此选项可用于指定一个或多个要与之隔离的CPU通用的SMP平衡和调度算法.

因此，此选项将有效地防止调度程序将线程从一个内核迁移到竞争较少的另一个内核(SMP平衡).通常将isolcpus与pthread亲和力控制结合使用，以了解CPU布局的方式来固定线程，从而获得可预测的性能.

https://www.kernel.org/doc/Documentation/kernel-parameters.txt

-编辑-

好的，我明白你为什么感到困惑.是的，我个人会对此选项承担一致的行为.问题在于两个函数，select_task_rq_fair 和 select_task_rq_rt，它们负责选择新的 run_queue(本质上是选择在哪个 next_cpu 上运行).我对这两个函数进行了快速跟踪 (Systemtap)，对于 CFS，它总是会返回掩码中相同的第一个核心；对于RT，它将返回其他内核.我没有机会研究每种选择算法中的逻辑，但是您可以向Linux开发邮件列表中的维护者发送电子邮件以进行修复.

Desired behaviour: run a multi-threaded Linux program on a set of cores which have been isolated using isolcpus.

Here's a small program we can use as an example multi-threaded program:

#include <stdio.h>
#include <pthread.h>
#include <err.h>
#include <unistd.h>
#include <stdlib.h>

#define NTHR    16
#define TIME    60 * 5

void *
do_stuff(void *arg)
{
    int i = 0;

    (void) arg;
    while (1) {
        i += i;
        usleep(10000); /* dont dominate CPU */
    }
}

int
main(void)
{
    pthread_t   threads[NTHR];
    int     rv, i;

    for (i = 0; i < NTHR; i++) {
        rv = pthread_create(&threads[i], NULL, do_stuff, NULL);
        if (rv) {
            perror("pthread_create");
            return (EXIT_FAILURE);
        }
    }
    sleep(TIME);
    exit(EXIT_SUCCESS);
}

If I compile and run this on a kernel with no isolated CPUs, then the threads are spread out over my 4 CPUs. Good!

Now if I add isolcpus=2,3 to the kernel command line and reboot:

• Running the program without taskset distributes threads over cores 0 and 1. This is expected as the default affinity mask now excludes cores 2 and 3.
• Running with taskset -c 0,1 has the same effect. Good.
• Running with taskset -c 2,3 causes all threads to go onto the same core (either core 2 or 3). This is undesired. Threads should distribute over cores 2 and 3. Right?

This post describes a similar issue (although the example given is farther away from the pthreads API). The OP was happy to workaround this by using a different scheduler. I'm not certain this is ideal for my use-case however.

Is there a way to have the threads distributed over the isolated cores using the default scheduler?

Is this a kernel bug which I should report?

EDIT:

The right thing does indeed happen if you use a real-time scheduler like the fifo scheduler. See man sched and man chrt for details.

解决方案

From the Linux Kernel Parameter Doc:

This option can be used to specify one or more CPUs to isolate from the general SMP balancing and scheduling algorithms.

So this options would effectively prevent scheduler doing thread migration from one core to another less contended core (SMP balancing). As typical isolcpus are used together with pthread affinity control to pin threads with knowledge of CPU layout to gain predictable performance.

https://www.kernel.org/doc/Documentation/kernel-parameters.txt

--Edit--

Ok I see why you are confused. Yeah personally I would assume consistent behavior on this option. The problem lies around two functions, select_task_rq_fair and select_task_rq_rt, which is responsible for selecting new run_queue (which is essentially selecting which next_cpu to run on). I did a quick trace (Systemtap) of both functions, for CFS it would always return the same first core in the mask; for RT, it would return other cores. I haven't got a chance to look into the logic in each selection algorithm but you can send an email to the maintainer in Linux devel mailing list for fix.

这篇关于为什么使用任务集在一组隔离的内核上运行多线程Linux程序会导致所有线程在一个内核上运行?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！