为每个MPI进程分配不同数量的OpenMP线程 [英] Assign different number of openmp threads to each mpi process

问题描述

假设我有一个在384个MPI进程(24个计算节点，每个计算节点具有16个核心)上运行的代码，并使用以下简单脚本将我的作业提交到作业队列中

Assume that I have a code that runs on 384 MPI processes (24 compute nodes with 16 cores per compute node) and use the following simple script to submit my job to a job queue

#!/bin/bash
#PBS -S /bin/bash
#PBS -l nodes=24:ppn=16
#PBS -l walltime=01:00:00

cd $PBS_O_WORKDIR
module load openmpi
mpirun mycode > output_file

是否可能出现以下情况:我需要再分配一个具有16个内核的节点，以使用'openmp'进行某些特定的计算，并在某些时候用计算结果更新384个进程的其余部分.因此，现在我有384个MPI进程，每个进程按顺序运行一个线程，而一个MPI进程有16个openmp线程.

Is the following scenario possible: I need to assign one more node with 16 cores to do some specific calculations using 'openmp' and updates the rest of the 384 processes at some point with the results of the computations. So now I have 384 MPI processes with one thread running sequentially on each and one MPI process with 16 openmp threads.

是否可以通过OMP_NUM_THREADS和mpirun或任何其他工具来完成此任务?

Is it possible to accomplish this by OMP_NUM_THREADS and mpirun or any other tools?

我感谢任何建议

谢谢

新浪

推荐答案

您可以请求16个ppns的25个节点，然后仅强制385个MPI进程:

You could request 25 nodes with 16 ppns and then force only 385 MPI processes:

#PBS -l nodes=25:ppn=16
...
mpirun -np 384 mycode : -np 1 -x OMP_NUM_THREADS=16 mycode > output_file

这利用Open MPI的MPMD启动模式，并以冒号分隔不同的启动配置.由于默认情况下，等级是按顺序填充到节点插槽上的，因此前384个等级将精确地跨越24个节点，然后其他等级将在最后一个节点上开始.为此，OMP_NUM_THREADS环境变量将被设置为16，因此启用了16个OpenMP线程.如果OpenMP程序是其他可执行文件，则只需在第二个启动配置中替换其名称即可，例如:

This utilises the MPMD launch mode of Open MPI with different launch configurations separated by colons. Since by default ranks are populated sequentially over node slots, the first 384 ranks will span exactly 24 nodes, then the additional rank will get started on the very last node. For it the OMP_NUM_THREADS environment variable will get set to 16 therefore enabling 16 OpenMP threads. If the OpenMP program is a different executable, just substitute its name in the second launch configuration, e.g.:

mpirun -np 384 mycode : -np 1 -x OMP_NUM_THREADS=16 myompcode > output_file

这篇关于为每个MPI进程分配不同数量的OpenMP线程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！