确保混合MPI/OpenMP在不同的内核上运行每个OpenMP线程 [英] Ensure hybrid MPI / OpenMP runs each OpenMP thread on a different core

问题描述

我正在尝试运行混合的OpenMP/MPI作业，以使OpenMP线程被内核分开(每个内核仅一个线程).我看到了其他使用numa-ctl和bash脚本设置环境变量的答案，我不想这样做.

I am trying to get a hybrid OpenMP / MPI job to run so that OpenMP threads are separated by core (only one thread per core). I have seen other answers which use numa-ctl and bash scripts to set environment variables, and I don't want to do this.

我只想通过在命令行上设置OMP_NUM_THREADS和或OMP_PROC_BIND和mpiexec选项来执行此操作.我尝试了以下方法-假设我想要2个MPI进程，每个进程都有2个OpenMP线程，并且每个线程都在单独的内核上运行，所以我总共希望有4个内核.

I would like to be able to do this only by setting OMP_NUM_THREADS and or OMP_PROC_BIND and mpiexec options on the command line. I have tried the following - let's say I want 2 MPI processes that each have 2 OpenMP threads, and each of the threads are run on separate cores, so I want 4 cores total.

OMP_PROC_BIND=true OMP_PLACES=cores OMP_NUM_THREADS=2 mpiexec -n 2 

这将拆分作业，以便只有两个进程在工作，并且它们都在同一CPU上，因此它们每个仅占用大约25％的CPU.如果我尝试:

This splits the jobs so that only two processes are at work, and they are all on the same CPU, so they are each only using about 25% of the CPU. If I try:

OMP_PROC_BIND=false OMP_PLACES=cores OMP_NUM_THREADS=2 mpiexec -n 2

然后，我得到了两个单独的MPI进程，每个进程都以其CPU能力的100％或100％以上运行.这似乎并未显示出用于OpenMP线程的不同内核.

then, I just get two separate MPI processes, each running at 100% or over 100% of their CPU power, according to top. This doesn't seem to show different cores being used for OpenMP threads.

如何强制系统在单独的内核上放置单独的线程?

How do I force the system to put separate threads on separate cores?

仅供参考，lscpu会显示以下内容:

FYI, lscpu prints this:

-CPU(s):                48
-On-line CPU(s) list:   0-47
-Thread(s) per core:    2
-Core(s) per socket:    12
-Socket(s):             2
-NUMA node(s):          2

推荐答案

实际上，我希望您的第一个示例能够奏效.在此处设置OMP_PROC_BIND=true很重要，这样，在钉住线程的线程时，OpenMP会保留在MPI进程与CPU的绑定中.

Actually, I'd expect your first example to work. Setting the OMP_PROC_BIND=true here is important, so that OpenMP stays within the CPU binding from the MPI process when pinning it's threads.

取决于批处理系统和MPI实施，可能会有非常单独的方法来设置这些内容.

Depending on the batch system and MPI implementation, there might be very individual ways to set these things up.

在Linux中，所有超线程或通常每个内核有多个硬件线程都显示为内核"可能是问题的一部分，因为当两个进程在两个超线程上运行时，您将永远看不到200％一个核心.

Also Hyperthreading, or in general multiple hardware threads per core, that all show up as "cores" in your Linux, might be part of the problem as you'll never see 200% when two processes run on the two Hyperthreads of one cores.

这是一个通用的解决方案，我在为某些MPI和某些系统上的某些OpenMP实现确定这些东西时会用到. Cray提供了一个文档，其中包含一个非常有用的程序，可以快速找出这些问题，称为xthi.c，

Here is a generic solution, I use when figuring these things for some MPI and some OpenMP implementation on some system. There's documentation from Cray which contains a very helpful program to figure these things out quickly, it's called xthi.c, google the filename or paste it from here (not sure if it's legal to paste it here...). Compile with:

mpicc xthi.c -fopenmp -o xthi

现在我们可以看到发生了什么，例如，在具有超线程和Intel MPI(基于MPICH)的2x 8 Core Xeon上，我们得到了:

Now we can see what exactly is going on, for instance on a 2x 8 Core Xeon with Hyperthreading and Intel MPI (MPICH-based) we get:

$ OMP_PROC_BIND=true OMP_PLACES=cores OMP_NUM_THREADS=2 mpiexec -n 2 ./xthi

Hello from rank 0, thread 0, on localhost. (core affinity = 0,16)
Hello from rank 0, thread 1, on localhost. (core affinity = 1,17)
Hello from rank 1, thread 0, on localhost. (core affinity = 8,24)
Hello from rank 1, thread 1, on localhost. (core affinity = 9,25)

如您所见，核心是指核心的所有超线程.请注意，默认情况下，mpirun也是如何将其固定在不同的插座上的.而使用OMP_PLACES=threads，每个内核可以获得一个线程:

As you can see, core means, all the Hyperthreads of a core. Note how mpirun pins it different sockets, too by default. And With OMP_PLACES=threads you get one thread per core:

$ OMP_PROC_BIND=true OMP_PLACES=threads OMP_NUM_THREADS=2 mpiexec -n 2 ./xthi
Hello from rank 0, thread 0, on localhost. (core affinity = 0)
Hello from rank 0, thread 1, on localhost. (core affinity = 1)
Hello from rank 1, thread 0, on localhost. (core affinity = 8)
Hello from rank 1, thread 1, on localhost. (core affinity = 9)

使用OMP_PROC_BIND=false(第二个示例)，我得到:

With OMP_PROC_BIND=false (your second example), I get:

$ OMP_PROC_BIND=false OMP_PLACES=cores OMP_NUM_THREADS=2 mpiexec -n 2 ./xthi
Hello from rank 0, thread 0, on localhost. (core affinity = 0-7,16-23)
Hello from rank 0, thread 1, on localhost. (core affinity = 0-7,16-23)
Hello from rank 1, thread 0, on localhost. (core affinity = 8-15,24-31)
Hello from rank 1, thread 1, on localhost. (core affinity = 8-15,24-31)

在这里，每个OpenMP线程都有一个完整的套接字，因此MPI等级仍在不同的资源上运行.但是，操作系统可以在所有内核中疯狂地调度一个进程中的OpenMP线程.就像在测试系统上设置OMP_NUM_THREADS=2一样.

Here, each OpenMP thread gets a full socket, so the MPI ranks still operate on distinct resources. However, the OpenMP threads, within one process could be scheduled wildly by the OS across all cores. It's the same as just setting OMP_NUM_THREADS=2 on my test system.

同样，这可能取决于特定的OpenMP和MPI实现和版本，但是我认为您可以很容易理解上面的描述.

Again, this might depend on specific OpenMP and MPI implementations and versions, but I think you'll easily figure out what's going on with the description above.

希望有帮助.

这篇关于确保混合MPI/OpenMP在不同的内核上运行每个OpenMP线程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！