MPI Fortran代码：如何通过openMP共享节点上的数据？ [英] MPI Fortran code: how to share data on node via openMP?

问题描述

现在，我正面临一种情况，一组数据增长非常大，但对于每个进程，所以我宁愿将它存储在内存中，每个节点只有一次，并且一个节点上的所有进程访问相同的数据。

为每个进程存储一次超过可用的RAM。

是否有可能通过openMP实现类似的功能？

数据共享节点是我想要的唯一东西，不需要其他每个节点都需要并行化，因为这已经通过MPI完成了。

解决方案

如果仅用于共享大块数据，则不需要实施混合MPI + OpenMP代码。你实际上必须做的是：1）将世界通信器分成跨越相同主机/节点的组。如果你的MPI库实现了MPI-3.0，那真的很简单 - 你需要做的就是调用 MPI_COMM_SPLIT_TYPE 和 split_type set至 MPI_COMM_TYPE_SHARED ：

  USE mpi_f08 
 
 TYPE（MPI_Comm）:: hostcomm 
 
 CALL MPI_Comm_split_type（MPI_COMM_WORLD，MPI_COMM_TYPE_SHARED，0，& 
 MPI_INFO_NULL，hostcomm）
  

MPI-2.2或更早版本不提供 MPI_COMM_SPLIT_TYPE 操作，而且必须具备一定的创造性。例如，您可以使用我的简单拆分实现，可以在Github上找到此处< a）。
$ b $ 2）现在驻留在同一个节点上的进程是同一个通信器 hostcomm 的一部分，他们可以创建一个共享内存块并用它来交换数据。再次，MPI-3.0提供了一种（相对）简单且便携的方式来实现这一点：

  USE mpi_f08 
 USE， INTRINSIC :: ISO_C_BINDING，ONLY：C_PTR，C_F_POINTER 
 
 INTEGER :: hostrank 
 
 INTEGER（KIND = MPI_ADDRESS_KIND）:: size 
 INTEGER :: disp_unit 
 TYPE（C_PTR）:: baseptr 
 TYPE（MPI_Win）:: win 
 
 TYPE（MY_DATA_TYPE），POINTER :: shared_data 
 
！我们只需要每个主机一个进程分配内存
！在所有进程中将大小设置为0，但是一个
 CALL MPI_Comm_rank（hostcomm，hostrank）
 if（hostrank == 0）then 
 size = 10000000！把实际数据大小放在这里
 else 
 size = 0 
 end if 
 disp_unit = 1 
 CALL MPI_Win_allocate_shared（size，disp_unit，MPI_INFO_NULL，& 
 hostcomm，baseptr，win）
 
！获取内存段的位置
 if（hostrank / = 0）then 
 CALL MPI_Win_shared_query（win，0，size，disp_unit，baseptr）
 end if 
 
 ！ baseptr现在可以与Fortran指针
关联！并因此用于访问共享数据
 CALL C_F_POINTER（baseptr，shared_data）
 
！使用shared_data就好像它是ALLOCATE'd 
！ ... 
 
！销毁共享内存窗口
 CALL MPI_Win_free（win）
  

代码的工作方式是它使用MPI-3.0功能来分配共享内存窗口。 MPI_WIN_ALLOCATE_SHARED 在每个进程中分配一块共享内存。由于您想共享一个数据块，因此仅在一个进程中分配它并且不会将其分配到各个进程中，因此 size 被设置为0只有一个人在打电话时排队。 MPI_WIN_SHARED_QUERY 用于找出共享内存块映射到调用进程的虚拟地址空间的地址。一个地址是已知的，C指针可以使用 C_F_POINTER（）子例程与Fortran指针相关联，后者可用于访问共享内存。一旦完成，必须通过使用 MPI_WIN_FREE 销毁共享内存窗口来释放共享内存。

MPI-2.2或更早版本不提供共享内存窗口。在这种情况下，必须使用操作系统相关的API来创建共享内存块，例如，标准POSIX序列 shm_open（） / ftruncate（） / mmap（）。必须编写可从Fortran调用的实用程序C函数才能执行这些操作。请参阅该代码以获取灵感。由 mmap（）返回的 void * 可以直接传递到Fortran代码中 C_PTR 类型变量，然后可以与Fortran指针关联。

I am working on an Fortan code that already uses MPI.

Now, I am facing a situation, where a set of data grows very large but is same for every process, so I would prefer to store it in memory only once per node and all processes on one node access the same data.

Storing it once for every process would go beyond the available RAM.

Is it somehow possible to achieve something like that with openMP?

Data sharing per node is the only thing I would like to have, no other per node paralellisation required, because this is already done via MPI.

解决方案

You don't need to implement a hybrid MPI+OpenMP code if it is only for sharing a chunk of data. What you actually have to do is:

1) Split the world communicator into groups that span the same host/node. That is really easy if your MPI library implements MPI-3.0 - all you need to do is call MPI_COMM_SPLIT_TYPE with split_type set to MPI_COMM_TYPE_SHARED:

USE mpi_f08

TYPE(MPI_Comm) :: hostcomm

CALL MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0, &
                         MPI_INFO_NULL, hostcomm)

MPI-2.2 or earlier does not provide the MPI_COMM_SPLIT_TYPE operation and one has to get somewhat creative. You could for example use my simple split-by-host implementation that can be found on Github here.

2) Now that processes that reside on the same node are part of the same communicator hostcomm, they can create a block of shared memory and use it to exchange data. Again, MPI-3.0 provides an (relatively) easy and portable way to do that:

USE mpi_f08
USE, INTRINSIC :: ISO_C_BINDING, ONLY : C_PTR, C_F_POINTER

INTEGER :: hostrank

INTEGER(KIND=MPI_ADDRESS_KIND) :: size
INTEGER :: disp_unit
TYPE(C_PTR) :: baseptr
TYPE(MPI_Win) :: win

TYPE(MY_DATA_TYPE), POINTER :: shared_data

! We only want one process per host to allocate memory
! Set size to 0 in all processes but one
CALL MPI_Comm_rank(hostcomm, hostrank)
if (hostrank == 0) then
   size = 10000000 ! Put the actual data size here
else
   size = 0
end if
disp_unit = 1
CALL MPI_Win_allocate_shared(size, disp_unit, MPI_INFO_NULL, &
                             hostcomm, baseptr, win)

! Obtain the location of the memory segment
if (hostrank /= 0) then
   CALL MPI_Win_shared_query(win, 0, size, disp_unit, baseptr)
end if

! baseptr can now be associated with a Fortran pointer
! and thus used to access the shared data
CALL C_F_POINTER(baseptr, shared_data)

! Use shared_data as if it was ALLOCATE'd
! ...

! Destroy the shared memory window
CALL MPI_Win_free(win)

The way that code works is that it uses the MPI-3.0 functionality for allocating shared memory windows. MPI_WIN_ALLOCATE_SHARED allocates a chunk of shared memory in each process. Since you want to share one block of data, it only makes sense to allocate it in a single process and not have it spread across the processes, therefore size is set to 0 for all but one ranks while making the call. MPI_WIN_SHARED_QUERY is used to find out the address at which that shared memory block is mapped in the virtual address space of the calling process. One the address is known, the C pointer can be associated with a Fortran pointer using the C_F_POINTER() subroutine and the latter can be used to access the shared memory. Once done, the shared memory has to be freed by destroying the shared memory window with MPI_WIN_FREE.

MPI-2.2 or earlier does not provide shared memory windows. In that case one has to use the OS-dependent APIs for creation of shared memory blocks, e.g. the standard POSIX sequence shm_open() / ftruncate() / mmap(). A utility C function callable from Fortran has to be written in order to perform those operations. See that code for some inspiration. The void * returned by mmap() can be passed directly to the Fortran code in a C_PTR type variable that can be then associated with a Fortran pointer.

这篇关于MPI Fortran代码：如何通过openMP共享节点上的数据？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！