绑定线程到某些MPI进程 [英] binding threads to certain MPI processes

问题描述

我有以下设置，一个混合MPI / OpenMP代码，
运行M个MPI进程，每个进程有N个线程。总共有
MxN线程可用。

如果可能，我想要做的是仅将线程
分配给某些MPI进程而不是全部，我的代码将为
更有效，因为有些线程只是做
重复工作。

谢谢。

解决方案

您的问题是这一个。至少有三种可能的解决方案。

对于大多数MPI实现，可以使用他们自己的环境（上下文）启动多个可执行文件作为相同MPI作业的一部分。它被称为MPMD（多程序多数据）或MIMD（多指令多数据）模型。语法通常包含：（冒号）作为分隔符：

  $ mpiexec<全局参数> 
 -n n1<本地参数> executable_1< args1> ：
 -n n2<本地参数> executable_2< args2> ：
 ... 
 -n nk<本地参数> executable_k< argsk> 
  

它启动 n1 等级运行 executable_1 带有命令行参数< args1> ， n2 executable_2 带有命令行参数< args2> ，依此类推。总共 n1 + n2 + ... + nk 过程开始，行列被线性分配：

 排名（从..到）|可执行
 ==================== | ============= 
 0 .. n1-1 | executable_1 
 n1 .. n1 + n2-1 | executable_2 
 n1 + n2 .. n1 + n2 + n3-1 | executable_3 
 ... | ... 
  

作为更狭义的情况，可以指定相同的可执行文件 k 次，以便使用相同的可执行文件获取不同的上下文。 < local parameters> 可以包括设置特定环境变量的值，例如，在你的情况下，可能是 OMP_NUM_THREADS 。指定环境的确切方法因实现而异。使用Open MPI，可以这样做：

  mpiexec --hostfile all_hosts \ 
 -n 5 -x OMP_NUM_THREADS = 2 myprog：\ 
 -n 4 -x OMP_NUM_THREADS = 4 myprog：\ 
 -n 6 -x OMP_NUM_THREADS = 1 myprog 
  

这将在 all_hosts （全局参数）中指定的主机上启动15个MPI等级。前五个使用两个OpenMP线程，接下来的四个OpenMP线程和最后六个按顺序运行。使用基于MPICH的实现时，该命令会稍有不同：

$ b

  mpiexec --hostfile all_hosts \ 
 -n 5 -env OMP_NUM_THREADS 2 myprog：\ 
 -n 4 -env OMP_NUM_THREADS 4 myprog：\ 
 -n 6 -env OMP_NUM_THREADS 1 myprog 
  

虽然得到广泛支持，但以前的方法有点不灵活。如果人们想要例如所有队伍除了连续第10次跑？然后命令行变成：

  mpiexec ... 
 -n 9 -x OMP_NUM_THREADS = 1 myprog：\ 
 -n 1 -x OMP_NUM_THREADS = N myprog：\ 
 -n 9 -x OMP_NUM_THREADS = 1 myprog：\ 
 -n 1 -x OMP_NUM_THREADS = N myprog：\ 
 ... 
  

更方便的解决方案是提供基于进程级别设置 OMP_NUM_THREADS 的包装器。例如，Open MPI的这种封装看起来像：

$ b

 ＃！/ bin / bash 
 if [$（（（$ OMPI_COMM_WORLD_RANK + 1）％10））== 0];然后
 export OMP_NUM_THREADS = N 
 else 
 export OMP_NUM_THREADS = 1 
 fi 
 exec$ *
  

，用法如下：

  mpiexec -n M ... mywrapper.sh myprog< args> 
  

第三个也是最不灵活的选项是简单地调用 omp_set_num_threads（） code>在MPI初始化之后，但在任何并行区域之前，并根据等级设置不同的线程数：



调用MPI_COMM_RANK（MPI_COMM_WORLD，等级，ierr）$ b $ integer $ ::



$ b if（mod（rank，10）== 0）then
call omp_set_num_threads（N）
else
call omp_set_num_threads（1）
end if


不管选择什么解决方案，进程和线程绑定都变得有点棘手，应该完全关闭。 / p>

I have the following setup, a hybrid MPI/OpenMP code which runs M MPI processes with N threads each. In total there are MxN threads available.

What I would like to do, if possible, is to assign threads only to some MPI processes not to all of them, my code would be more efficient since some of the threads are just doing repetitive work.

Thanks.

解决方案

Your question is a generalised version of this one. There are at least three possible solutions.

With most MPI implementations it is possible to start multiple executables with their own environments (contexts) as part of the same MPI job. It is called MPMD (Multiple Programs Multiple Data) or MIMD (Multiple Instructions Multiple Data) model. The syntax usually involves : (colon) as a separator:

$ mpiexec <global parameters>
          -n n1 <local parameters> executable_1 <args1> :
          -n n2 <local parameters> executable_2 <args2> :
          ...
          -n nk <local parameters> executable_k <argsk>

It launches n1 ranks running executable_1 with command-line arguments <args1>, n2 ranks running executable_2 with command-line arguments <args2>, and so on. In total n1 + n2 + ... + nk processes are started and ranks are assigned linearly:

 Ranks (from .. to) |  Executable
====================|=============
0     .. n1-1       | executable_1
n1    .. n1+n2-1    | executable_2
n1+n2 .. n1+n2+n3-1 | executable_3
...                 | ...

As a more narrow case, the same executable could be specified k times in order to get k different contexts with the same executable. <local parameters> could include setting the values of specific environment variables, e.g. in your case that could be OMP_NUM_THREADS. The exact method to specify the environment differs from one implementation to another. With Open MPI, one would do:

mpiexec --hostfile all_hosts \
        -n 5 -x OMP_NUM_THREADS=2 myprog : \
        -n 4 -x OMP_NUM_THREADS=4 myprog : \
        -n 6 -x OMP_NUM_THREADS=1 myprog

That will start 15 MPI ranks on the hosts specified in all_hosts (a global parameters) with the first five using two OpenMP threads, the next four - four OpenMP threads, and the last six running sequentially. With MPICH-based implementations the command would be slightly different:

mpiexec --hostfile all_hosts \
        -n 5 -env OMP_NUM_THREADS 2 myprog : \
        -n 4 -env OMP_NUM_THREADS 4 myprog : \
        -n 6 -env OMP_NUM_THREADS 1 myprog

Although widely supported, the previous method is a bit inflexible. What if one would like e.g. all ranks except every 10-th run sequentially? Then the command line becomes:

mpiexec ...
        -n 9 -x OMP_NUM_THREADS=1 myprog : \
        -n 1 -x OMP_NUM_THREADS=N myprog : \
        -n 9 -x OMP_NUM_THREADS=1 myprog : \
        -n 1 -x OMP_NUM_THREADS=N myprog : \
        ...

A more convenient solution would be to provide a wrapper that sets OMP_NUM_THREADS based on the process rank. For example, such a wrapper for Open MPI looks like:

#!/bin/bash
if [ $((($OMPI_COMM_WORLD_RANK + 1) % 10)) == 0 ]; then
  export OMP_NUM_THREADS=N
else
  export OMP_NUM_THREADS=1
fi
exec "$*"

and is used simply as:

mpiexec -n M ... mywrapper.sh myprog <args>

The third and least flexible option is to simply call omp_set_num_threads() from within the program after MPI initialisation but before any parallel regions and set different number of threads based on the rank:

integer :: provided, rank, ierr

call MPI_INIT_THREAD(MPI_THREAD_FUNNELED, provided, ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)

if (mod(rank, 10) == 0) then
   call omp_set_num_threads(N)
else
   call omp_set_num_threads(1)
end if

No matter what solution is chosen, process and thread binding becomes a bit tricky and should probably be switched off altogether.

这篇关于绑定线程到某些MPI进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！