MPI_Type_create_subarray 和 MPI_Gather [英] MPI_Type_create_subarray and MPI_Gather
问题描述
我必须解决一个小问题.我有 4 个从进程，每个进程都想发送一个二维子数组 (CHUNK_ROWS X CHUNK_COLUMNS) 到主 0.主 0 收集 ddd[ROWS][COLUMNS] 中的所有块并打印它.我想使用 MPI_Gather()
#include #include 使用命名空间标准；#define 行 10#define 列 10#define CHUNK_ROWS 5#define CHUNK_COLUMNS 5#定义标签0int** alloca_matrice(int righe，int colone){int** 矩阵=NULL；国际我;矩阵 = (int **)malloc(righe * sizeof(int*));如果(矩阵！= NULL){矩阵[0] = (int *)malloc(righe*colonne*sizeof(int));如果(矩阵[0]！=NULL)for(i=1; i 谢谢大家.
所以这有点微妙，需要了解 Gather 集合如何放置复杂类型.
如果您查看大多数 MPI_Gather 的示例，它们是一维数组，很容易解释应该发生什么；你从每个进程中得到(比如)10 个整数，而 Gather 足够聪明，可以将第 0 级的 10 个整数放在数组中的第 10-19 位，以此类推.
不过，像这样更复杂的布局要复杂一些.首先，发送方的数据布局不同于接收方的数据布局.从发送者的角度来看，你从数组元素 [1][2] 开始，转到 [1][5](在一个大小为 7x7 的数组中)，然后跳转到数组元素[2][3]-[2][5]等.有 CHUNK_ROWS 数据块，每个数据块由 2 个整数分隔.
现在考虑接收者必须如何接收它们.假设它正在接收 rank 0 的数据.它将接收到数组元素 [0][0]-[0][4] —— 到目前为止一切顺利；但随后它将接收下一个数据块到 [1][0]-[1][4] 中，大小为 10x10 的数组.那是跳过 5 个元素.内存中的布局是不同的.因此接收方必须接收到不同的 Subarray 类型然后发送方发送，因为内存布局不同.
因此，虽然您可能从如下所示的内容发送:
尺寸[0] = CHUNK_ROWS+2;尺寸[1] = CHUNK_COLUMNS+2;subsizes[0] = CHUNK_ROWS;子尺寸[1] = CHUNK_COLUMNS;开始[0] = 1;开始[1] = 1;MPI_Type_create_subarray(ndims,sizes,subsizes,starts,MPI_ORDER_C,MPI_INT,&sendsubarray);MPI_Type_commit(&sendsubarray);您将收到如下所示的内容:
尺寸[0] = ROWS;尺寸[1] = 列；subsizes[0] = CHUNK_ROWS;子尺寸[1] = CHUNK_COLUMNS;开始[0] = 0;开始[1] = 0;MPI_Type_create_subarray(ndims,sizes,subsizes,starts,MPI_ORDER_C,MPI_INT,&recvsubarray);MPI_Type_commit(&recvsubarray);最重要的是，注意 sizes 数组的不同.
现在我们越来越近了.请注意您的 MPI_Gather 行更改为如下所示:
MPI_Gather(DEBUG_CH[0],1,sendsubarray,recvptr,1,recvsubarray,0,MPI_COMM_WORLD);有几件事对以前的版本不起作用，MPI_Gather(DEBUG_CH[0],1,subarray,ddd[0],CHUNK_ROWS*CHUNK_COLUMNS,MPI_INT,0,MPI_COMM_WORLD); -- 首先，请注意您正在引用 ddd[0]，但对于除 rank 0 之外的每个等级，ddd=NULL，因此这将失败.因此，创建一个名为 say recvptr 的新变量，并在零级中将其设置为 ddd[0].(其他进程认为它在哪里并不重要，因为它们没有接收.)另外，我认为您不想接收 CHUNK_ROWS*CHUNK_COLUMS MPI_INTs，因为这会将它们连续放置在内存中，我的理解是您希望它们以与工作任务相同的方式排列，但在更大的数组中.
好的，现在我们到达了某个地方，但上面的仍然不起作用，这是一个有趣的原因.对于一维数组示例，很容易找出第 n 行数据的去向.它的计算方式是找到接收到的数据的范围，然后开始下一个元素.但这在这里行不通.就在"之后排名零数据的末尾不是排名数据应该开始的地方 ([0][5]) 而是 [4][5] -- 后面的元素秩为 0 的子数组中的最后一个元素.在这里，您从不同级别收到的数据重叠！所以我们将不得不摆弄数据类型的范围，并手动指定每个等级的数据开始的位置.第二个是容易的部分；您可以在需要时使用 MPI_Gatherv 功能手动指定来自每个处理器的数据量，或它的去向.第一个是比较棘手的部分.
MPI 让您指定给定数据类型的下限和上限——在给定一块内存的情况下，该类型的第一位数据将去哪里，以及它结束"的位置，这里仅表示下一个可以开始的地方.(数据可以超出类型的上限，我认为这会使这些名称产生误导，但事情就是这样.)您可以将其指定为您喜欢的任何方便您使用的东西；由于我们将处理 int 数组中的元素，让我们将类型的范围设为 MPI_INT 大小.
MPI_Type_create_resized(recvsubarray, 0, 1*sizeof(int), &resizedrevsubarray);MPI_Type_commit(&resizedrecvsubarray);(请注意，我们只需要对接收到的类型执行此操作；对于发送类型，因为我们只发送其中之一，所以没有关系).
现在，我们将使用 Gatherv 来指定每个元素的开始位置——以大小"为单位.这种新的调整大小的类型，它只是 1 个整数.所以如果我们想要一些东西进入[0][5]的大数组，从大数组开始的位移是5；如果我们想让它进入 [5][5] 位置，位移是 55.
最后，请注意，收集和分散集合都假设即使是主人"或协调器进程正在参与.如果即使协调器也拥有自己的全局数组部分，则最容易使其工作.
因此，以下对我有用:
#include #include #include 使用命名空间标准；#define 行 10#define 列 10#define CHUNK_ROWS 5#define CHUNK_COLUMNS 5#定义标签0int** alloca_matrice(int righe，int colone){int** 矩阵=NULL；国际我;矩阵 = (int **)malloc(righe * sizeof(int*));如果(矩阵！= NULL){矩阵[0] = (int *)malloc(righe*colonne*sizeof(int));如果(矩阵[0]！=NULL)for(i=1; i I have to solve a little mpi problem. I have 4 slaves processes and each of these wants to send a 2d subarray (CHUNK_ROWS X CHUNK_COLUMNS) to master 0. Master 0 collects all chunks in ddd[ROWS][COLUMNS] and print it. I want to use MPI_Gather()
#include <mpi.h>
#include <iostream>
using namespace std;
#define ROWS 10
#define COLUMNS 10
#define CHUNK_ROWS 5
#define CHUNK_COLUMNS 5
#define TAG 0
int** alloca_matrice(int righe, int colonne)
{
int** matrice=NULL;
int i;
matrice = (int **)malloc(righe * sizeof(int*));
if(matrice != NULL){
matrice[0] = (int *)malloc(righe*colonne*sizeof(int));
if(matrice[0]!=NULL)
for(i=1; i<righe; i++)
matrice[i] = matrice[0]+i*colonne;
else{
free(matrice);
matrice = NULL;
}
}
else{
matrice = NULL;
}
return matrice;
}
int main(int argc, char* argv[])
{
int my_id, numprocs,length,i,j;
int ndims, sizes[2],subsizes[2],starts[2];
int** DEBUG_CH=NULL;
int** ddd=NULL;
char name[BUFSIZ];
MPI_Datatype subarray=NULL;
//MPI_Status status;
MPI_Init(&argc, &argv) ;
MPI_Comm_rank(MPI_COMM_WORLD, &my_id) ;
MPI_Comm_size(MPI_COMM_WORLD, &numprocs) ; // Ottiene quanti processi sono attivi
MPI_Get_processor_name(name, &length);
if(my_id!=0){
//creo una sottomatrice ripulita dalle ghost cells
ndims=2;
sizes[0] = CHUNK_ROWS+2;
sizes[1] = CHUNK_COLUMNS+2;
subsizes[0] = CHUNK_ROWS;
subsizes[1] = CHUNK_COLUMNS;
starts[0] = 1;
starts[1] = 1;
MPI_Type_create_subarray(ndims,sizes,subsizes,starts,MPI_ORDER_C,MPI_INT,&subarray);
MPI_Type_commit(&subarray);
DEBUG_CH = alloca_matrice(CHUNK_ROWS+2,CHUNK_COLUMNS+2);
for(i=0; i<CHUNK_ROWS+2; i++){
for(j=0; j<CHUNK_COLUMNS+2; j++){
if(i==0 || i==CHUNK_ROWS+1 || j==0 || j==CHUNK_COLUMNS+1)
DEBUG_CH[i][j] = 5;
else
DEBUG_CH[i][j] = 1;
}
}
//MPI_Send(DEBUG_CH[0],1,subarray,0,TAG,MPI_COMM_WORLD);
}
if(my_id==0){
ddd = alloca_matrice(ROWS,COLUMNS);
}
MPI_Gather(DEBUG_CH[0],1,subarray,ddd[0],CHUNK_ROWS*CHUNK_COLUMNS,MPI_INT,0,MPI_COMM_WORLD);
if(!my_id){
for(i=0; i<ROWS; i++){
for(j=0; j<COLUMNS; j++){
printf("%d ",ddd[i][j]);
}
printf("
");
}
}
if(my_id)
MPI_Type_free(&subarray);
MPI_Finalize(); // Chiusura di MPI.
return 0;
}
Thanks all.
So this is a little more subtle, and requires some understanding of how the Gather collective places complex types.
If you look at most examples of MPI_Gather, they're of 1-d arrays, and it's fairly easy to interpret what should happen; you're getting (say) 10 ints from each process, and Gather is smart enough to put the 10 ints from rank 0 at the start, the 10 from rank 1 at positions 10-19 in the array, and so on.
More complex layouts like this are a little more complicated, though. First, the data layout from the sender's point of view is different from the data layout from the receivers. In the sender's point of view, you start at array element [1][2], go to [1][5] (in an array of size 7x7), then jump to array elements [2][3]-[2][5], etc.
There are CHUNK_ROWS blocks of data, each separated by 2 ints.
Now consider how the reciever has to receive them. Let's say it's receiving rank 0's data. It's going to receive that into array elements [0][0]-[0][4] -- so far so good; but then it's going to receive the next block of data into [1][0]-[1][4], in an array of size 10x10. That's a jump over 5 elements. The layout in memory is different. So the receiver will have to be recieving into a different Subarray type then the senders are sending from, because the memory layout is different.
So while you might be sending from something that looks like this:
sizes[0] = CHUNK_ROWS+2;
sizes[1] = CHUNK_COLUMNS+2;
subsizes[0] = CHUNK_ROWS;
subsizes[1] = CHUNK_COLUMNS;
starts[0] = 1;
starts[1] = 1;
MPI_Type_create_subarray(ndims,sizes,subsizes,starts,MPI_ORDER_C,MPI_INT,&sendsubarray);
MPI_Type_commit(&sendsubarray);
you'll be receiving into something that looks like this:
sizes[0] = ROWS;
sizes[1] = COLUMNS;
subsizes[0] = CHUNK_ROWS;
subsizes[1] = CHUNK_COLUMNS;
starts[0] = 0; starts[1] = 0;
MPI_Type_create_subarray(ndims,sizes,subsizes,starts,MPI_ORDER_C,MPI_INT,&recvsubarray);
MPI_Type_commit(&recvsubarray);
Crucially, notice the difference in the sizes array.
Now we're getting a little closer. Notice your MPI_Gather line changes to something like this:
MPI_Gather(DEBUG_CH[0],1,sendsubarray,recvptr,1,recvsubarray,0,MPI_COMM_WORLD);
There were a couple things that didn't work about the previous version, MPI_Gather(DEBUG_CH[0],1,subarray,ddd[0],CHUNK_ROWS*CHUNK_COLUMNS,MPI_INT,0,MPI_COMM_WORLD); -- first, note that you're referencing ddd[0], but for every rank except rank 0, ddd=NULL, and so this will fail. So create a new variable called say recvptr, and in rank zero, set that to ddd[0]. (It doesn't matter where the other processes think it is, as they're not receiving.) Also, I think you don't want to be receiving CHUNK_ROWS*CHUNK_COLUMS MPI_INTs, because that would place them contiguously in memory, and my understanding is you want them layed out the same way as on the worker tasks, but in the larger array.
Ok, so now we're getting somewhere, but the above still won't work, for an interesting reason. For the 1d array examples, it's easy enough to figure out where the nth ranks data goes. The way it's calculated is by finding the extent of the data being recieved, and starting the next element just after that one. But that won't work here. "Just after" the end of rank zero's data is not where rank one's data should start ([0][5]) but instead, [4][5] -- the element after the last element in rank 0s subarray. Here, the data you're reciving from different ranks overlaps! So we're going to have to fiddle with the extents of the data types, and manually specify where each rank's data starts. The second is the easy part; you use the MPI_Gatherv function when you need to manually specify the amount of data from each processor, or where it goes. The first is the trickier part.
MPI let's you specify the lower and upper bounds of a given data type -- where, given a piece of memory, the first bit of data for this type would go, and where it "ends", which here only means where the next one could start. (The data can extend past the upper bound of the type, which I'd argue makes those names misleading, but such is the way of things.) You can specify this to be anything you like that makes it convenient for you; since we'll be dealing with elements in an int array, let's make the extent of our type one MPI_INT in size.
MPI_Type_create_resized(recvsubarray, 0, 1*sizeof(int), &resizedrevsubarray);
MPI_Type_commit(&resizedrecvsubarray);
(Note we only have to do this for the received type; from the send type, since we're only sending one of them, it doesn't matter).
Now, we'll use gatherv to specify where each element starts -- in units of the "size" of this new resized type, which is just 1 integer. So if we want something to go into the large array at [0][5], the displacement from the start of the large array is 5; if we want it to go in there at position [5][5], the displacement is 55.
Finally, notice that the gather and scatter collectives all assume that even the "master" or coordinator process is participating. It's easiest to get this working if even the coordinator has their own piece of the global array.
So with that, the following works for me:
#include <mpi.h>
#include <iostream>
#include <cstdlib>
using namespace std;
#define ROWS 10
#define COLUMNS 10
#define CHUNK_ROWS 5
#define CHUNK_COLUMNS 5
#define TAG 0
int** alloca_matrice(int righe, int colonne)
{
int** matrice=NULL;
int i;
matrice = (int **)malloc(righe * sizeof(int*));
if(matrice != NULL){
matrice[0] = (int *)malloc(righe*colonne*sizeof(int));
if(matrice[0]!=NULL)
for(i=1; i<righe; i++)
matrice[i] = matrice[0]+i*colonne;
else{
free(matrice);
matrice = NULL;
}
}
else{
matrice = NULL;
}
return matrice;
}
int main(int argc, char* argv[])
{
int my_id, numprocs,length,i,j;
int ndims, sizes[2],subsizes[2],starts[2];
int** DEBUG_CH=NULL;
int** ddd=NULL;
int *recvptr=NULL;
char name[BUFSIZ];
MPI_Datatype sendsubarray;
MPI_Datatype recvsubarray;
MPI_Datatype resizedrecvsubarray;
//MPI_Status status;
MPI_Init(&argc, &argv) ;
MPI_Comm_rank(MPI_COMM_WORLD, &my_id) ;
MPI_Comm_size(MPI_COMM_WORLD, &numprocs) ; // Ottiene quanti processi sono attivi
if (numprocs != 4) {
MPI_Abort(MPI_COMM_WORLD,1);
}
MPI_Get_processor_name(name, &length);
//creo una sottomatrice ripulita dalle ghost cells
ndims=2;
sizes[0] = CHUNK_ROWS+2;
sizes[1] = CHUNK_COLUMNS+2;
subsizes[0] = CHUNK_ROWS;
subsizes[1] = CHUNK_COLUMNS;
starts[0] = 1;
starts[1] = 1;
MPI_Type_create_subarray(ndims,sizes,subsizes,starts,MPI_ORDER_C,MPI_INT,&sendsubarray);
MPI_Type_commit(&sendsubarray);
DEBUG_CH = alloca_matrice(CHUNK_ROWS+2,CHUNK_COLUMNS+2);
for(i=0; i<CHUNK_ROWS+2; i++){
for(j=0; j<CHUNK_COLUMNS+2; j++){
if(i==0 || i==CHUNK_ROWS+1 || j==0 || j==CHUNK_COLUMNS+1)
DEBUG_CH[i][j] = 5;
else
DEBUG_CH[i][j] = my_id;
}
}
recvptr=DEBUG_CH[0];
if(my_id==0){
ddd = alloca_matrice(ROWS,COLUMNS);
sizes[0] = ROWS; sizes[1] = COLUMNS;
subsizes[0] = CHUNK_ROWS; subsizes[1] = CHUNK_COLUMNS;
starts[0] = 0; starts[1] = 0;
MPI_Type_create_subarray(2,sizes,subsizes,starts,MPI_ORDER_C,MPI_INT,&recvsubarray);
MPI_Type_commit(&recvsubarray);
MPI_Type_create_resized(recvsubarray, 0, 1*sizeof(int), &resizedrecvsubarray);
MPI_Type_commit(&resizedrecvsubarray);
recvptr = ddd[0];
}
int counts[5]={1,1,1,1};
int disps[5] ={0,5,50,55};
MPI_Gatherv(DEBUG_CH[0],1,sendsubarray,recvptr,counts,disps,resizedrecvsubarray,0,MPI_COMM_WORLD);
if(!my_id){
for(i=0; i<ROWS; i++){
for(j=0; j<COLUMNS; j++){
printf("%d ",ddd[i][j]);
}
printf("
");
}
}
if(my_id == 0) {
MPI_Type_free(&resizedrecvsubarray);
MPI_Type_free(&recvsubarray);
free(ddd[0]);
free(ddd);
} else {
MPI_Type_free(&sendsubarray);
free(DEBUG_CH[0]);
free(DEBUG_CH);
}
MPI_Finalize(); // Chiusura di MPI.
return 0;
}
这篇关于MPI_Type_create_subarray 和 MPI_Gather的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
