使用并行NetCDF保存分布式3D复杂阵列 [英] Using parallel NetCDF to save a distributed 3D complex array
问题描述
我有一个用Fortran编写的基于MPI的程序，该程序在每个节点(2D时间序列的各个部分)上生成复杂数据的3D数组.我想使用并行I/O将这些数组写入单个文件，可以在python中相对容易地打开该文件以进行进一步的分析/可视化.理想情况下，我希望该解决方案能够提高内存效率(即避免创建中间临时数组).
I have an MPI-based program written in Fortran which produces a 3D array of complex data at each node (sections of a 2D time-series). I would like to use parallel I/O to write these arrays to a single file which can be relatively easily opened in python for further analysis/visualization. Ideally I would like the solution to be memory efficient (i.e. avoid the creation of intermediate temporary arrays).
使用NetCDF，我设法调整了子例程可以实现3D实数数组.但是，对于复杂数组，我遇到了绊脚石.
Using NetCDF, I have managed to adapt a subroutine which achieves this for a 3D array of real numbers. However, I've hit a stumbling block when it comes to complex arrays.
在下面的代码中，我试图通过创建由两个实数组成的复合数据类型，并假设Fortran复杂数据类型的实部和虚部部分连续存储在第一个维度上，来将子例程从实数扩展为复数.3D阵列.
In the following code I have attempted to extend the subroutine from reals to complex numbers by creating a compound datatype consisting of two reals and assuming that the real and imaginary components of the Fortran complex datatype are stored contiguously in the 1st dimension of the 3D array.
module IO
use NetCDF
use, intrinsic :: iso_fortran_env, only: dp => real64
implicit none
contains
subroutine output_3D(dataname, starts, ends, global_data_dims, &
local_data, MPI_communicator)
character(len=*), intent(in) :: dataname
integer, dimension(3), intent(in) :: starts
integer, dimension(3), intent(in) :: ends
integer, dimension(3), intent(in) :: global_data_dims
complex(dp), intent(in) :: local_data( 1:(ends(1) - starts(1)+ 1), &
1:(ends(2) - starts(2) + 1), &
1:(ends(3) - starts(3) + 1))
integer, dimension(3) :: expanded_starts
integer, intent(in) :: MPI_communicator
integer :: ncid, varid, dimid(3)
integer :: counts(3)
integer :: typeid
expanded_starts(1) = (starts(1))* 2 + 1
expanded_starts = starts(2)
expanded_starts(3) = starts(3)
call check(nf90_create( trim(dataname)//'.cdf', &
IOR(NF90_NETCDF4, NF90_MPIIO), &
ncid, &
comm = MPI_communicator, &
info = MPI_INFO_NULL))
call check(nf90_def_dim(ncid, "x", global_data_dims(1), dimid(1)))
call check(nf90_def_dim(ncid, "y", global_data_dims(2) * 2, dimid(2)))
call check(nf90_def_dim(ncid, "z", global_data_dims(3), dimid(3)))
! define a complex data type consisting of two real(8)
call check(nf90_def_compound(ncid, 16, "COMPLEX", typeid))
call check(nf90_insert_compound(ncid, typeid, "REAL", 0, NF90_DOUBLE))
call check(nf90_insert_compound(ncid, typeid, "IMAG", 8, NF90_DOUBLE))
! define a 3D variable of type "complex"
call check(nf90_def_var(ncid, dataname, typeid, dimid, varid))
! exit define mode
call check(nf90_enddef(ncid))
! Now in NETCDF data mode
! set to use MPI/PnetCDF collective I/O
call check(nf90_var_par_access(ncid, varid, NF90_COLLECTIVE))
counts(1) = (ends(1) - starts(1) + 1) * 2
counts(2) = (ends(2) - starts(2) + 1)
counts(3) = (ends(3) - starts(3) + 1)
call check(nf90_put_var(ncid, &
varid, &
local_data, &
start = expanded_starts,&
count = counts))
! close the file
call check(nf90_close(ncid))
return
end subroutine output_3D
subroutine check(status)
integer, intent ( in) :: status
if(status /= nf90_noerr) then
print *, trim(nf90_strerror(status))
stop 2
end if
end subroutine check
end module IO
program test_write
use IO
use MPI
complex(dp) :: data(2,2,3)
integer :: flock
integer :: rank
integer :: ierr
integer :: i, j, k
call MPI_init(ierr)
call MPI_comm_size(MPI_comm_world, flock, ierr)
call MPI_comm_rank(MPI_comm_world, rank, ierr)
do k = 1, 3
do j = 1, 2
do i = 1, 2
data(i,j,k) = cmplx(i, j, 8)
enddo
enddo
enddo
if (rank == 0) then
call output_3D_hdf5('out', [1,1,1], [2,2,3], [2,2,6], &
data, MPI_comm_world)
else
call output_3D_hdf5('out', [1,1,4], [2,2,6], [2,2,6], &
data, MPI_comm_world)
endif
call MPI_finalize(ierr)
end program test_write
以上代码导致编译时出现"nf90_put_var没有特定功能"错误.这表明该函数对输入数组的数据类型不满意，因此很明显我在使用复合数据类型方面缺少一些东西.
The above code results in an "There is no specific function for nf90_put_var" error on compilation. This indicates that the function is not happy with the data type of the input array, so clearly there is something I'm missing regarding the usage of compound data types.
One simple workaround is to assign the complex array to a real pointer as described in this post. The array can then be reshaped/recast using numpy to arrive at the complex array in python. It's a bit clunky, and somewhat dissatisfying - but is probably good enough for my purposes for now.
推荐答案
出于以下原因，这仅是部分答案-注释太长.希望我能够找到丢失的信息并升级"，但这是我到目前为止的信息.
This is only a partial answer for reasons you will see below - but it is too long for a comment. Hopefully I will be able to find the missing info and "upgrade" it, but this is what I have so far.
如果您在 要以复合类型写入数据，请首先使用 To write data in a compound type, first use 请注意，它根本没有提到 Note it doesn't mention 到目前为止，您应该调用 So far so good, you should call one of 尽管我在这里确实不应该为变量的种类使用显式数字，但我可以向您展示 While I am here you really shouldn't use explicit numbers for kinds for variables, I can show you compilers where 这篇关于使用并行NetCDF保存分布式3D复杂阵列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！ nf90_def_compound 创建类型，多次调用 nf90_insert_compound 添加到复合类型，然后使用适当的 nf90_put_var1 ， nf90_put_vara ， nf90_put_vars 或 nf90_put_varm调用.
nf90_def_compound to create the type, multiple calls to nf90_insert_compound to add to the compound type, and then write data with the appropriate nf90_put_var1, nf90_put_vara, nf90_put_vars, or nf90_put_varm call. nf90_put_var ，而是4种不同的功能.这在某种意义上说， nf90_put_var 可能很好地重载了NetCDF支持的所有内在类型(并且完全废弃，它不支持复杂的)，因此对于非内在类型，大概有一些类似于C的接口，带有 void * 之类的东西，我猜这是上述四个函数实现的.nf90_put_var at all, but 4 different functions. This makes some kind of sense, nf90_put_var is presumably nicely overloaded to deal with all intrinsic types NetCDF support (and it is utterly crap it doesn't support complex), so for non-intrinsic type presumably there is some C like interface with something like a void *, and I'm guessing it is this that the four functions mentioned above implement. nf90_put_var1 ， nf90_put_vara ， nf90_put_vars 或 nf90_put_varm 之一比 nf90_put_var .现在是个坏消息-我找不到这4个功能的任何文档.等效的C函数位于 https://www.unidata.ucar.edu/software/netcdf/docs/group__variables.html ，因此您也许可以从那里算出所需的内容，但这并不是很好-我至少会向Unidata提交错误报告，但这对我来说，缺乏对复杂性的内在支持足以让我在其他地方寻找我的I/O解决方案... nf90_put_var1, nf90_put_vara, nf90_put_vars, or nf90_put_varm rather than nf90_put_var. Now the bad news - I can't find any documentation for these 4 functions. The equivalent C functions are at https://www.unidata.ucar.edu/software/netcdf/docs/group__variables.html so you might be able to work out what is required from there, but it's not very good - I'd at the very least put in a bug report to Unidata, but that said for me lack of intrinsic support for complex is enough to make me look elsewhere for my I/O solution ... complex(8)无法编译的编译器.取而代之的是使用 Selected_real_kind 或类似的代码，或者使用固有模块iso_fortran_env中的常数，或者可能使用iso_c_binding中的常数，并且复数的种类与构成实数的种类相同的事实它.complex(8) will fail to compile. Instead use Selected_real_kind or similar, or use the constant in the intrinsic module iso_fortran_env, or possibly those in iso_c_binding, and the fact that the kind of a complex number is the same as the kind of the reals that compose it.
