MPI Fortran 代码:如何通过 openMP 在节点上共享数据? [英] MPI Fortran code: how to share data on node via openMP?

问题描述

我正在编写一个已经使用 MPI 的 Fortan 代码.

I am working on an Fortan code that already uses MPI.

现在，我面临一种情况，一组数据增长得非常大，但每个进程都相同，所以我希望每个节点只将它存储在内存中一次，并且一个节点上的所有进程都访问相同的数据.

Now, I am facing a situation, where a set of data grows very large but is same for every process, so I would prefer to store it in memory only once per node and all processes on one node access the same data.

为每个进程存储一次会超出可用 RAM.

Storing it once for every process would go beyond the available RAM.

是否有可能使用 openMP 实现类似的目标?

Is it somehow possible to achieve something like that with openMP?

每个节点的数据共享是我唯一想要的，不需要其他每个节点的并行化，因为这已经通过 MPI 完成.

Data sharing per node is the only thing I would like to have, no other per node paralellisation required, because this is already done via MPI.

推荐答案

如果只是为了共享一块数据，你不需要实现混合 MPI+OpenMP 代码.你实际上要做的是:

You don't need to implement a hybrid MPI+OpenMP code if it is only for sharing a chunk of data. What you actually have to do is:

1) 将世界通信器分成跨越同一主机/节点的组.如果您的 MPI 库实现 MPI-3.0，那真的很容易 - 您只需调用 MPI_COMM_SPLIT_TYPE 并将 split_type 设置为 MPI_COMM_TYPE_SHARED:p>

1) Split the world communicator into groups that span the same host/node. That is really easy if your MPI library implements MPI-3.0 - all you need to do is call MPI_COMM_SPLIT_TYPE with split_type set to MPI_COMM_TYPE_SHARED:

USE mpi_f08

TYPE(MPI_Comm) :: hostcomm

CALL MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0, &
                         MPI_INFO_NULL, hostcomm)

MPI-2.2 或更早版本不提供 MPI_COMM_SPLIT_TYPE 操作，因此必须有点创意.例如，您可以使用我在 Github here.

MPI-2.2 or earlier does not provide the MPI_COMM_SPLIT_TYPE operation and one has to get somewhat creative. You could for example use my simple split-by-host implementation that can be found on Github here.

2) 现在驻留在同一节点上的进程是同一通信器hostcomm 的一部分，它们可以创建一个共享内存块并使用它来交换数据.同样，MPI-3.0 提供了一种(相对)简单且可移植的方式来做到这一点:

2) Now that processes that reside on the same node are part of the same communicator hostcomm, they can create a block of shared memory and use it to exchange data. Again, MPI-3.0 provides an (relatively) easy and portable way to do that:

USE mpi_f08
USE, INTRINSIC :: ISO_C_BINDING, ONLY : C_PTR, C_F_POINTER

INTEGER :: hostrank

INTEGER(KIND=MPI_ADDRESS_KIND) :: size
INTEGER :: disp_unit
TYPE(C_PTR) :: baseptr
TYPE(MPI_Win) :: win

TYPE(MY_DATA_TYPE), POINTER :: shared_data

! We only want one process per host to allocate memory
! Set size to 0 in all processes but one
CALL MPI_Comm_rank(hostcomm, hostrank)
if (hostrank == 0) then
   size = 10000000 ! Put the actual data size here
else
   size = 0
end if
disp_unit = 1
CALL MPI_Win_allocate_shared(size, disp_unit, MPI_INFO_NULL, &
                             hostcomm, baseptr, win)

! Obtain the location of the memory segment
if (hostrank /= 0) then
   CALL MPI_Win_shared_query(win, 0, size, disp_unit, baseptr)
end if

! baseptr can now be associated with a Fortran pointer
! and thus used to access the shared data
CALL C_F_POINTER(baseptr, shared_data)

! Use shared_data as if it was ALLOCATE'd
! ...

! Destroy the shared memory window
CALL MPI_Win_free(win)

代码的工作方式是使用 MPI-3.0 功能来分配共享内存窗口.MPI_WIN_ALLOCATE_SHARED 在每个进程中分配一块共享内存.由于您要共享一个数据块，因此仅将其分配在单个进程中而不是将其分布在各个进程中才有意义，因此 size 设置为 0 为除了一个等级之外的所有等级通话.MPI_WIN_SHARED_QUERY 用于找出该共享内存块在调用进程的虚拟地址空间中映射的地址.一个地址是已知的，C 指针可以使用 C_F_POINTER() 子例程与 Fortran 指针相关联，后者可用于访问共享内存.完成后，必须通过使用 MPI_WIN_FREE 销毁共享内存窗口来释放共享内存.

The way that code works is that it uses the MPI-3.0 functionality for allocating shared memory windows. MPI_WIN_ALLOCATE_SHARED allocates a chunk of shared memory in each process. Since you want to share one block of data, it only makes sense to allocate it in a single process and not have it spread across the processes, therefore size is set to 0 for all but one ranks while making the call. MPI_WIN_SHARED_QUERY is used to find out the address at which that shared memory block is mapped in the virtual address space of the calling process. One the address is known, the C pointer can be associated with a Fortran pointer using the C_F_POINTER() subroutine and the latter can be used to access the shared memory. Once done, the shared memory has to be freed by destroying the shared memory window with MPI_WIN_FREE.

MPI-2.2 或更早版本不提供共享内存窗口.在这种情况下，必须使用依赖于操作系统的 API 来创建共享内存块，例如标准 POSIX 序列 shm_open()/ftruncate()/mmap().为了执行这些操作，必须编写一个可从 Fortran 调用的实用 C 函数.请参阅该代码以获得一些灵感.mmap() 返回的 void * 可以直接传递给 C_PTR 类型变量中的 Fortran 代码，然后可以将其与 Fortran 关联指针.

MPI-2.2 or earlier does not provide shared memory windows. In that case one has to use the OS-dependent APIs for creation of shared memory blocks, e.g. the standard POSIX sequence shm_open() / ftruncate() / mmap(). A utility C function callable from Fortran has to be written in order to perform those operations. See that code for some inspiration. The void * returned by mmap() can be passed directly to the Fortran code in a C_PTR type variable that can be then associated with a Fortran pointer.

这篇关于MPI Fortran 代码:如何通过 openMP 在节点上共享数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！