如何:带Infiniband的Azure OpenMPI-Linux [英] How to: Azure OpenMPI with Infiniband - Linux

问题描述

我不熟悉将Microsoft Azure用于科学计算的目的，并且在设置过程中遇到了一些问题.

I am new to using Microsoft Azure for scientific computing purposes and have encountered a few issues whilst setting up.

我有一个跳箱设置，可以用作我希望使用的软件的许可证服务器，并且还具有一个通用驱动器来存储所有软件. 还设置了6个计算节点(16个核心/节点)，我可以毫无问题地从跳转框"ssh"到计算节点.跳转框和计算节点将CentOS与OpenMPI 1.10.3一起使用

I have a jump box set-up that acts as a license server for the software that I whish to use, is also has a common drive to store all of the software. 6 compute nodes are also set-up (16 core/node) and I can 'ssh' from the jump box to the compute nodes without issue. The jump box and compute nodes are using CentOS with OpenMPI 1.10.3

我已经创建了一个脚本，该脚本存储在我通过"clusRun.sh"在每个计算节点上运行的已安装跳箱驱动器上，该脚本设置了特定于我运行的软件和OpenMPI的所有环境变量. 希望这一切听起来不错.

I have created a script that is stored on the mounted jump box drive that I run on each compute node through 'clusRun.sh' which sets up all the environment variable specific to the software I run and OpenMPI. Hopefully it all sounds good to this point.

过去，我已经在Linux集群上使用了很多此软件，而没有出现任何问题.作业是使用类似如下的命令提交的:

I've used this software on Linux clusters a lot in the past without issue. The jobs are submitted using a command similar such as:

mpirun -np XXX -hostfile XXX {软件路径}

mpirun -np XXX -hostfile XXX {path to software}

其中XXX是处理器数量和主机文件路径

Where XXX is the number of processors and path to hostfile

我在跳转框上运行此命令，并且主机文件包含每个计算节点名称的列表，每个计算节点名称在主机文件中的次数与我希望在该节点上的内核的次数相同.希望有道理！作业没有在跳转框节点上运行的进程，它仅用于启动作业.

I run this command on the jump box and the hostfile has a list of the names of each compute node, each compute node name is in the hostfile the same number of times as cores I want on the node. Hope that makes sense! There are no processes from the job running on the jump box node, it's merely used to launch the job.

当我尝试以这种方式运行作业时，我收到许多错误，大多数似乎与Infiniband捆绑在一起.以下是关键错误的简短列表:

When I try and run the jobs this way, I receive a number of errors, most seem to be tied up with Infiniband. Here is a short list of the key errors:

"The /dev/hfi1_0 device failed to appear after 15.0 seconds: Connection timed out"

"The OpenFabrics (openib) BTL failed to initialize while trying to create an internal queue"

"OMPI source: btl_openib.c:324
 Function: ibv_create_srq()
 Error: Function not implemented (errno=38)
 Device: mlx4_0"

至少一对MPI进程无法相互进行MPI通信.这意味着没有开放的MPI设备指示它可以用于这些进程之间的通信"

"At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes"

是否需要设置特定于OpenMPI的任何定义Infiniband设置的环境变量?我已经定义了通常的MPI_BIN，LD_LIBRARY_PATH，PATH等.我知道IntelMPI需要其他变量.

Are there any environment variables specific to OpenMPI that need to be set-up that define any Infiniband settings? I have already defined the usual MPI_BIN, LD_LIBRARY_PATH, PATH etc. I know that IntelMPI requires additional variables.

Infiniband应该作为A9 HPC分配的一部分出现，但是我不确定是否需要任何特定的设置.当我运行'ifconfig -a'时，没有Infiniband特定的条目(我希望看到ib0，ib1等).我只有eth0，eth1和lo

The Infiniband should come as part of the A9 HPC allocation, however I'm not sure if it need any specific setting up. When I run 'ifconfig -a' there are no Infiniband specific entries (I expect to see ib0, ib1 etc). I just have eth0, eth1 and lo

我希望有人能提供任何建议.

I look forward to any advise that someone might be able to offer.

亲切的问候！

推荐答案

如包含的存储库中所述由daltskin 撰写的关于azure.microsoft.com文档中心的文章"(从已删除/隐藏的 Azure/azure-content-internal )，在第

As stated in "Repository containing the Articles on azure.microsoft.com Documentation Center" by daltskin (forked from deleted/hidden Azure/azure-content-internal), at page https://github.com/daltskin/azure-content/blob/master/articles/virtual-machines/virtual-machines-a8-a9-a10-a11-specs.md#access-to-the-rdma-network "About the A8, A9, A10, and A11 compute-intensive instances" - "Access from Linux A8 and A9 VMs"

当前，仅Intel MPI Library 5支持Azure Linux RDMA.

At this time, Azure Linux RDMA is supported only with Intel MPI Library 5.

因此，带有OpenMPI 1.10.3的CentOS可能不适用于Azure的此虚拟RDMA，因为OpenMPI 1.10.3不是英特尔MPI库5".

So, CentOS with OpenMPI 1.10.3 probably will not work with this virtualized RDMA by Azure, as OpenMPI 1.10.3 is not the "Intel MPI Library 5".

在官方文档中，Azure还将英特尔MPI列为支持RDMA的(使用SLES 12 SP1 HPC VM): https://docs.microsoft.com/zh-cn/azure/virtual-machines/linux/classic/rdma-cluster 设置Linux RDMA群集以运行MPI应用程序"-2017-3-14

In official docs Azure lists Intel MPI as RDMA-enabled too (with SLES 12 SP1 HPC VM): https://docs.microsoft.com/en-us/azure/virtual-machines/linux/classic/rdma-cluster "Set up a Linux RDMA cluster to run MPI applications" - 2017-3-14

自定义VM

Customize the VM

在SLES 12 SP1 HPC VM中，建议您不要应用内核更新，这可能会导致Linux RDMA驱动程序出现问题. 英特尔MPI:通过运行以下命令，在SLES 12 SP1 HPC VM上完成英特尔MPI的安装:

In a SLES 12 SP1 HPC VM, we recommend that you don't apply kernel updates, which can cause issues with the Linux RDMA drivers. Intel MPI: Complete the installation of Intel MPI on the SLES 12 SP1 HPC VM by running the following command:

 sudo rpm -v -i --nodeps /opt/intelMPI/intel_mpi_packages/*.rpm

如果要基于Azure市场中基于CentOS的HPC映像之一而不是SLES 12 for HPC来建立群集，请按照上一节中的常规步骤进行操作.在配置和配置VM时，请注意以下差异: 通过基于CentOS的HPC映像配置的VM上已经安装了Intel MPI.

If you want to set up a cluster based on one of the CentOS-based HPC images in the Azure Marketplace instead of SLES 12 for HPC, follow the general steps in the preceding section. Note the following differences when you provision and configure the VM: Intel MPI is already installed on a VM provisioned from a CentOS-based HPC image.

因此，有Azure虚拟RDMA(Infiniband)的专有内核驱动程序，已从Azure预安装到SLES 12 VM映像中，并已从Azure预安装到CentOS VM映像中，还有专有的用户空间驱动程序(因为Infiniband通常使用内核旁路和对话功能)只能在Intel MPI中从用户空间进行数据移动操作).

So, there is proprietary kernel driver of Azure virtual RDMA (Infiniband), preinstalled into SLES 12 VM image from Azure and into CentOS VM image from Azure, and also proprietary user-space driver (as Infiniband commonly uses kernel bypass and talk to hw from user-space for data movement operations) only in Intel MPI.

尝试使用预安装的Intel MPI重新编译您的应用程序，并使用Intel MPI的mpirun/mpiexec启动它.指令仍在同一 https://docs.microsoft.com/zh-CN/azure/virtual-machines/linux/classic/rdma-cluster :

Try to recompile your application with preinstalled Intel MPI and start it with Intel MPI's mpirun/mpiexec. Instruction is still on the same https://docs.microsoft.com/en-us/azure/virtual-machines/linux/classic/rdma-cluster:

配置英特尔MPI要在Azure Linux RDMA上运行MPI应用程序，您需要 需要配置特定于英特尔MPI的某些环境变量. 这是一个示例Bash脚本，用于配置运行所需的变量 一个应用程序.根据需要将路径更改为mpivars.sh 安装英特尔MPI.

Configure Intel MPI To run MPI applications on Azure Linux RDMA, you need to configure certain environment variables specific to Intel MPI. Here is a sample Bash script to configure the variables needed to run an application. Change the path to mpivars.sh as needed for your installation of Intel MPI.

#!/bin/bash -x

# For a SLES 12 SP1 HPC cluster

source /opt/intel/impi/5.0.3.048/bin64/mpivars.sh

# For a CentOS-based HPC cluster

# source /opt/intel/impi/5.1.3.181/bin64/mpivars.sh

export I_MPI_FABRICS=shm:dapl

# THIS IS A MANDATORY ENVIRONMENT VARIABLE AND MUST BE SET BEFORE RUNNING ANY JOB
# Setting the variable to shm:dapl gives best performance for some applications
# If your application doesn’t take advantage of shared memory and MPI together, then set only dapl

export I_MPI_DAPL_PROVIDER=ofa-v2-ib0

# THIS IS A MANDATORY ENVIRONMENT VARIABLE AND MUST BE SET BEFORE RUNNING ANY JOB

export I_MPI_DYNAMIC_CONNECTION=0

# THIS IS A MANDATORY ENVIRONMENT VARIABLE AND MUST BE SET BEFORE RUNNING ANY JOB

# Command line to run the job

mpirun -n <number-of-cores> -ppn <core-per-node> -hostfile <hostfilename>  /path <path to the application exe> <arguments specific to the application>

#end

这篇关于如何:带Infiniband的Azure OpenMPI-Linux的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！