使用MPI发送二维数组块用C [英] sending blocks of 2D array in C using MPI

问题描述

你怎么发送2-D阵列块不同的处理器？假设二维数组的大小是400×400的我想给大小100X100块给不同的处理器。其思想是每个处理器将其单独的块上执行的计算及其结果发送回对最终结果的第一处理器。结果
我使用MPI在C程序中。

How do you send blocks of 2-D array to different processors? Suppose the 2D array size is 400x400 an I want to send blocks of sizes 100X100 to different processors. The idea is that each processor will perform computation on its separate block and send its result back to the first processor for final result.
I am using MPI in C programs.

推荐答案

让我说，你通常并不真的想这样做，开始 - 散射和一些大师的过程收集数据的大块。通常情况下，你希望每个任务被隆隆走在自己的一块拼图，你的目标应该是永远不会有一个处理器需要整个数据的全球视野;只要你需要，你限制了可扩展性和问题的规模。如果你这样做是对I / O - 一个进程读取数据，然后驱散它，然后收集回来写作，你会希望最终寻找到MPI-IO

Let me start by saying that you generally don't really want to do this - scatter and gather huge chunks of data from some "master" process. Normally you want each task to be chugging away at its own piece of the puzzle, and you should aim to never have one processor need a "global view" of the whole data; as soon as you require that, you limit scalability and the problem size. If you're doing this for I/O - one process reads the data, then scatters it, then gathers it back for writing, you'll want eventually to look into MPI-IO.

获取你的问题，但是，MPI具有拉动任意数据的内存不足的非常好的方式，而且分散/收集它，并从一组处理器。不幸的是，需要有相当数量的MPI概念 - MPI类型，程度和集体行动。很多的基本思路是在回答这个问题的讨论 - <一个href=\"http://stackoverflow.com/questions/5585630/mpi-type-create-subarray-and-mpi-gather\">MPI_Type_create_subarray和MPI_Gather 的。

Getting to your question, though, MPI has very nice ways of pulling arbitrary data out of memory, and scatter/gathering it to and from a set of processors. Unfortunately that requires a fair number of MPI concepts - MPI Types, extents, and collective operations. A lot of the basic ideas are discussed in the answer to this question -- MPI_Type_create_subarray and MPI_Gather .

更新 - 在冷光源的一天，这是很多code，而不是很多的解释。因此，让我扩大一点。

Update - In the cold light of day, this is a lot of code and not a lot of explanation. So let me expand a little bit.

考虑一个1D整数全球阵列任务0有要分发到一些MPI任务，让他们分别获得在当地阵列一块。假设你有4个任务，和全球阵列 [01234567] 。你可以有任务0发送四条消息（包括一个本身）分发此，当它的时间来重新组装，收到四条消息捆绑它重新走到一起;但显然变得非常时间大量消耗的过程。有这几类业务的优化程序 - 分散/收集操作。因此，在这种情况下，1D你会做这样的事情：

Consider a 1d integer global array that task 0 has that you want to distribute to a number of MPI tasks, so that they each get a piece in their local array. Say you have 4 tasks, and the global array is [01234567]. You could have task 0 send four messages (including one to itself) to distribute this, and when it's time to re-assemble, receive four messages to bundle it back together; but that obviously gets very time consuming at large numbers of processes. There are optimized routines for these sorts of operations - scatter/gather operations. So in this 1d case you'd do something like this:

int global[8];   /* only task 0 has this */
int local[2];    /* everyone has this */
const int root = 0;   /* the processor with the initial global data */

if (rank == root) {
   for (int i=0; i<7; i++) global[i] = i;
}

MPI_Scatter(global, 2, MPI_INT,      /* send everyone 2 ints from global */
            local,  2, MPI_INT,      /* each proc receives 2 ints into local */
            root, MPI_COMM_WORLD);   /* sending process is root, all procs in */
                                     /* MPI_COMM_WORLD participate */

在此之后，处理器的数据看起来像

After this, the processors' data would look like

task 0:  local:[01]  global: [01234567]
task 1:  local:[23]  global: [garbage-]
task 2:  local:[45]  global: [garbage-]
task 3:  local:[67]  global: [garbage-]

即，在分散操作需要全局数组，并发送连续的2-INT块的所有处理器。

That is, the scatter operation takes the global array and sends contiguous 2-int chunks to all the processors.

要重新组装阵列，我们使用了 MPI_Gather（）操作，其工作方式完全一样，但在相反的：

To re-assemble the array, we use the MPI_Gather() operation, which works exactly the same but in reverse:

for (int i=0; i<2; i++) 
   local[i] = local[i] + rank;

MPI_Gather(local,  2, MPI_INT,      /* everyone sends 2 ints from local */
           global, 2, MPI_INT,      /* root receives 2 ints each proc into global */
           root, MPI_COMM_WORLD);   /* recv'ing process is root, all procs in */
                                    /* MPI_COMM_WORLD participate */

和现在的数据看起来像

task 0:  local:[01]  global: [0134679a]
task 1:  local:[34]  global: [garbage-]
task 2:  local:[67]  global: [garbage-]
task 3:  local:[9a]  global: [garbage-]

收集带来所有的数据后面，这里是10，因为我没有在启动这个例子想我的格式通过精心就够了。

Gather brings all the data back, and here a is 10 because I didn't think my formatting through carefully enough upon starting this example.

如果数据点的数量不平均分配的进程数会发生什么，我们需要的项目不同的号码发送到每个进程？然后，你需要分散的通用版本， MPI_Scatterv（），它可以让你指定每个计数
处理器和位移 - 在那块数据开始全球数组中为止。因此，让我们说你有一个字符数组 [ABCDEFGHI] 9个字符，你要每一道工序两个字符指定除了最后一个，即有三个。然后，你需要

What happens if the number of data points doesn't evenly divide the number of processes, and we need to send different numbers of items to each process? Then you need a generalized version of scatter, MPI_Scatterv(), which lets you specify the counts for each processor, and displacements -- where in the global array that piece of data starts. So let's say you had an array of characters [abcdefghi] with 9 characters, and you were going to assign every process two characters except the last, that got three. Then you'd need

char global[9];   /* only task 0 has this */
char local[3]={'-','-','-'};    /* everyone has this */
int  mynum;                     /* how many items */
const int root = 0;   /* the processor with the initial global data */

if (rank == 0) {
   for (int i=0; i<8; i++) global[i] = 'a'+i;
}

int counts[4] = {2,2,2,3};   /* how many pieces of data everyone has */
mynum = counts[rank];
int displs[4] = {0,2,4,6};   /* the starting point of everyone's data */
                             /* in the global array */

MPI_Scatterv(global, counts, displs, /* proc i gets counts[i] pts from displs[i] */
            MPI_INT,      
            local, mynum, MPI_INT;   /* I'm receiving mynum MPI_INTs into local */
            root, MPI_COMM_WORLD);

现在数据的模样

task 0:  local:[ab-]  global: [abcdefghi]
task 1:  local:[cd-]  global: [garbage--]
task 2:  local:[ef-]  global: [garbage--]
task 3:  local:[ghi]  global: [garbage--]

您现在已经使用scatterv分发不规则的数据。在每种情况下的排量为两*等级（以字符为单位计量;排量在各类单位发送的散射或接收的聚集，它不是一般字节或东西）从数组的开始，而计数是{2,2,2,3}。如果它一直我们想有3个字符的第一个处理器，我们会设置数= {3,2,2,2}和位移本来{0,3,5,7}。 Gatherv再次工作完全一样，但扭转;计数和displs阵列将保持不变。

You've now used scatterv to distribute the irregular amounts of data. The displacement in each case is two*rank (measured in characters; the displacement is in unit of the types being sent for a scatter or received for a gather; it's not generally in bytes or something) from the start of the array, and the counts are {2,2,2,3}. If it had been the first processor we wanted to have 3 characters, we would have set counts={3,2,2,2} and displacements would have been {0,3,5,7}. Gatherv again works exactly the same but reverse; the counts and displs arrays would remain the same.

现在，对于2D，这是一个有点棘手。如果我们要发送一个二维数组的2D sublocks，我们现在可以发送已不再是数据是连续。如果我们将一个6x6的阵列（说）的3x3子块4个处理器，我们正在发送数据有孔的：

Now, for 2D, this is a bit trickier. If we want to send 2d sublocks of a 2d array, the data we're sending now no longer is contiguous. If we're sending (say) 3x3 subblocks of a 6x6 array to 4 processors, the data we're sending has holes in it:

2D Array

   ---------
   |000|111|
   |000|111|
   |000|111|
   |---+---|
   |222|333|
   |222|333|
   |222|333|
   ---------

Actual layout in memory

   [000111000111000111222333222333222333]

（注意所有高性能计算归结为理解在存储器中的数据的布局。）

(Note that all high-performance computing comes down to understanding the layout of data in memory.)

如果我们要发送标有1任务1，我们需要跳过三个值，送三个值，跳过三个值，送三个值，跳过三个值，送三个值中的数据。第二个难题是那里的分区域停止和启动;注意，区域1不启动，其中区域为0停止;区域为0的最后一个元素之后，在存储器中的下一个位置是通过区域1的中途单向

If we want to send the data that is marked "1" to task 1, we need to skip three values, send three values, skip three values, send three values, skip three values, send three values. A second complication is where the subregions stop and start; note that region "1" doesn't start where region "0" stops; after the last element of region "0", the next location in memory is partway-way through region "1".

让我们先解决第一个布局问题 - 如何拔出只是我们要发送的数据。我们总是可以只复制了所有的0的区域数据到另一个，连续数组，并发送;如果我们的计划出来不够仔细​​，我们甚至可以这样做，在这种我们可以称之为 MPI_Scatter 对结果的方法。但是，我们宁愿没有变调我们整个主要数据结构的方式。

Let's tackle the first layout problem first - how to pull out just the data we want to send. We could always just copy out all the "0" region data to another, contiguous array, and send that; if we planned it out carefully enough, we could even do that in such a way that we could call MPI_Scatter on the results. But we'd rather not have to transpose our entire main data structure that way.

到目前为止，我们已经使用了MPI数据类型是简单的 - MPI_INT指定（比如说）在连续4个字节。然而，MPI可以让你创建描述内存中任意复杂的数据布局自己的数据类型。而这种情况下 - 一个数组的矩形分区域 - 是很常见，有一个具体的要求这一点。为2维
情况下，我们上面描述的，

So far, all the MPI data types we've used are simple ones - MPI_INT specifies (say) 4 bytes in a row. However, MPI lets you create your own data types that describe arbitrarily complex data layouts in memory. And this case -- rectangular subregions of an array -- is common enough that there's a specific call for that. For the 2-dimensional case we're describing above,

    MPI_Datatype newtype;
    int sizes[2]    = {6,6};  /* size of global array */
    int subsizes[2] = {3,3};  /* size of sub-region */
    int starts[2]   = {0,0};  /* let's say we're looking at region "0",
                                 which begins at index [0,0] */

    MPI_Type_create_subarray(2, sizes, subsizes, starts, MPI_ORDER_C, MPI_INT, &newtype);
    MPI_Type_commit(&newtype);

此产生其中挑选出刚刚从全局阵列区域0的类型;我们可以
现在送只是一块数据到另一个处理器

This creates a type which picks out just the region "0" from the global array; we could send just that piece of data now to another processor

    MPI_Send(&(global[0][0]), 1, newtype, dest, tag, MPI_COMM_WORLD);  /* region "0" */

和接收过程中能接受它变成一个本地阵列。请注意，接收过程中，如果它只是接收它变成一个3x3的阵列，可以的不的描述一下它接受一个类型的 NEWTYPE ;不再描述存储器的布局。相反，它只是接收的3 * 3 = 9的整数块：

and the receiving process could receive it into a local array. Note that the receiving process, if it's only receiving it into a 3x3 array, can not describe what it's receiving as a type of newtype; that no longer describes the memory layout. Instead, it's just receiving a block of 3*3 = 9 integers:

    MPI_Recv(&(local[0][0]), 3*3, MPI_INT, 0, tag, MPI_COMM_WORLD);

请注意，我们可以为其他次区域，就此别过，通过创建不同类型的（具有不同的启动数组）的其他块，或只是通过发送在该特定块的起始点：

Note that we could do this for other sub-regions, too, either by creating a different type (with different start array) for the other blocks, or just by sending at the starting point of the particular block:

    MPI_Send(&(global[0][3]), 1, newtype, dest, tag, MPI_COMM_WORLD);  /* region "1" */
    MPI_Send(&(global[3][0]), 1, newtype, dest, tag, MPI_COMM_WORLD);  /* region "2" */
    MPI_Send(&(global[3][3]), 1, newtype, dest, tag, MPI_COMM_WORLD);  /* region "3" */

最后，请注意，我们需要全球和本地成为记忆在这里连续块;也就是说，及（全球[0] [0]）和及（本地[0] [0]）（或等价地， *国际和 *本地指向的连续6 * 6和3 * 3块存储器;未通过分配动态多-D阵列的通常的方式保证它示出了如何在下面做

Finally, note that we require global and local to be contiguous chunks of memory here; that is, &(global[0][0]) and &(local[0][0]) (or, equivalently, *global and *local point to contiguous 6*6 and 3*3 chunks of memory; that isn't guaranteed by the usual way of allocating dynamic multi-d arrays. It's shown how to do this below.

现在我们明白如何指定分区域，这里只有一件事使用分散/集中操作之前讨论，这就是在这些类型的大小。我们不能只使用 MPI_Scatter（）（甚至scatterv）这些类型的呢，因为这些类型的有16整数的程度;也就是说，在那里他们最终是16后的整数他们开始 - 以及他们最终没有下一个程序段开始的地方很好地排队，所以我们不能只用分散 - 它会选择错误的地方开始发送数据到下一个处理器

Now that we understand how to specify subregions, there's only one more thing to discuss before using scatter/gather operations, and that's the "size" of these types. We couldn't just use MPI_Scatter() (or even scatterv) with these types yet, because these types have an extent of 16 integers; that is, where they end is 16 integers after they start -- and where they end doesn't line up nicely with where the next block begins, so we can't just use scatter - it would pick the wrong place to start sending data to the next processor.

当然，我们可以使用 MPI_Scatterv（）并指定位移自己，这就是我们要做的 - 除了位移在发送型单位大小，并且不帮我们要么;块在（0,3,18,21）从全局数组开始偏移整数开始，一个块结束，从那里开始没有让我们的前preSS这些位移16整数事实整数倍的。

Of course, we could use MPI_Scatterv() and specify the displacements ourselves, and that's what we'll do - except the displacements are in units of the send-type size, and that doesn't help us either; the blocks start at offsets of (0,3,18,21) integers from the start of the global array, and the fact that a block ends 16 integers from where it starts doesn't let us express those displacements in integer multiples at all.

要解决这个问题，MPI可以设置类型的程度这些计算的目的。它不截断类型;它只是适用于那些下一个元素开始给出的最后一个元素搞清楚。对于类型，如这些有孔的，这是经常方便设置的程度是该类型的实际结束的东西比在存储器中的距离小。

To deal with this, MPI lets you set the extent of the type for the purposes of these calculations. It doesn't truncate the type; it's just used for figuring out where the next element starts given the last element. For types like these with holes in them, it's frequently handy to set the extent to be something smaller than the distance in memory to the actual end of the type.

我们可以设置在何种程度上是任何方便给我们。我们可以只让程度1的整数，然后设置位移以整数为单位。在这种情况下，虽然，我要设定的程度为3的整数 - 一个子行的大小 - 这样，块1的块0后，立即开始，块3块后立即启动 2。不幸的是，它并不完全来自块2跳楼来阻止3时的工作作为很好，但不能得到帮助。

We can set the extent to be anything that's convenient to us. We could just make the extent 1 integer, and then set the displacements in units of integers. In this case, though, I like to set the extent to be 3 integers - the size of a sub-row - that way, block "1" starts immediately after block "0", and block "3" starts immediately after block "2". Unfortunately, it doesn't quite work as nicely when jumping from block "2" to block "3", but that can't be helped.

于是撒子块在这种情况下，我们会做到以下几点：

So to scatter the subblocks in this case, we'd do the following:

    MPI_Datatype type, resizedtype;
    int sizes[2]    = {6,6};  /* size of global array */
    int subsizes[2] = {3,3};  /* size of sub-region */
    int starts[2]   = {0,0};  /* let's say we're looking at region "0",
                                 which begins at index [0,0] */

    /* as before */
    MPI_Type_create_subarray(2, sizes, subsizes, starts, MPI_ORDER_C, MPI_INT, &type);  
    /* change the extent of the type */
    MPI_Type_create_resized(type, 0, 3*sizeof(int), &resizedtype);
    MPI_Type_commit(&resizedtype);

下面我们创建相同的块型和以前一样，但我们已经重新调整它;我们没有改变其中类型开始（0），但我们已经改变了它目的（3整数）。我们没有提到过这一点，但 MPI_Type_commit 要求是能够使用类型;但你只需要提交您实际使用的最后一种，没有任何中间步骤。您可以使用 MPI_Type_free 来免费的，当你完成的类型。

Here we've created the same block type as before, but we've resized it; we haven't changed where the type "starts" (the 0) but we've changed where it "ends" (3 ints). We didn't mention this before, but the MPI_Type_commit is required to be able to use the type; but you only need to commit the final type you actually use, not any intermediate steps. You use MPI_Type_free to free the type when you're done.

所以，现在，我们终于可以scatterv块：以上数据操作有点复杂，但一旦它完成，scatterv看起来就像之前：

So now, finally, we can scatterv the blocks: the data manipulations above are a little complicated, but once it's done, the scatterv looks just like before:

int counts[4] = {1,1,1,1};   /* how many pieces of data everyone has, in units of blocks */
int displs[4] = {0,1,6,7};   /* the starting point of everyone's data */
                             /* in the global array, in block extents */

MPI_Scatterv(global, counts, displs, /* proc i gets counts[i] types from displs[i] */
            resizedtype,      
            local, 3*3, MPI_INT;   /* I'm receiving 3*3 MPI_INTs into local */
            root, MPI_COMM_WORLD);

现在，我们就大功告成了，分散了一点巡演之后，聚集，MPI派生类型。

And now we're done, after a little tour of scatter, gather, and MPI derived types.

这是例如code这表明无论是收集和分散操作，具有字符数组，如下。运行程序：

An example code which shows both the gather and the scatter operation, with character arrays, follows. Running the program:

$ mpirun -n 4 ./gathervarray
Global array is:
0123456789
3456789012
6789012345
9012345678
2345678901
5678901234
8901234567
1234567890
4567890123
7890123456
Local process on rank 0 is:
|01234|
|34567|
|67890|
|90123|
|23456|
Local process on rank 1 is:
|56789|
|89012|
|12345|
|45678|
|78901|
Local process on rank 2 is:
|56789|
|89012|
|12345|
|45678|
|78901|
Local process on rank 3 is:
|01234|
|34567|
|67890|
|90123|
|23456|
Processed grid:
AAAAABBBBB
AAAAABBBBB
AAAAABBBBB
AAAAABBBBB
AAAAABBBBB
CCCCCDDDDD
CCCCCDDDDD
CCCCCDDDDD
CCCCCDDDDD
CCCCCDDDDD

和code如下：

and the code follows.

#include <stdio.h>
#include <math.h>
#include <stdlib.h>
#include "mpi.h"

int malloc2dchar(char ***array, int n, int m) {

    /* allocate the n*m contiguous items */
    char *p = (char *)malloc(n*m*sizeof(char));
    if (!p) return -1;

    /* allocate the row pointers into the memory */
    (*array) = (char **)malloc(n*sizeof(char*));
    if (!(*array)) {
       free(p);
       return -1;
    }

    /* set up the pointers into the contiguous memory */
    for (int i=0; i<n; i++)
       (*array)[i] = &(p[i*m]);

    return 0;
}

int free2dchar(char ***array) {
    /* free the memory - the first element of the array is at the start */
    free(&((*array)[0][0]));

    /* free the pointers into the memory */
    free(*array);

    return 0;
}

int main(int argc, char **argv) {
    char **global, **local;
    const int gridsize=10; // size of grid
    const int procgridsize=2;  // size of process grid
    int rank, size;        // rank of current process and no. of processes

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);


    if (size != procgridsize*procgridsize) {
        fprintf(stderr,"%s: Only works with np=%d for now\n", argv[0], procgridsize);
        MPI_Abort(MPI_COMM_WORLD,1);
    }


    if (rank == 0) {
        /* fill in the array, and print it */
        malloc2dchar(&global, gridsize, gridsize);
        for (int i=0; i<gridsize; i++) {
            for (int j=0; j<gridsize; j++)
                global[i][j] = '0'+(3*i+j)%10;
        }


        printf("Global array is:\n");
        for (int i=0; i<gridsize; i++) {
            for (int j=0; j<gridsize; j++)
                putchar(global[i][j]);

            printf("\n");
        }
    }

    /* create the local array which we'll process */
    malloc2dchar(&local, gridsize/procgridsize, gridsize/procgridsize);

    /* create a datatype to describe the subarrays of the global array */

    int sizes[2]    = {gridsize, gridsize};         /* global size */
    int subsizes[2] = {gridsize/procgridsize, gridsize/procgridsize};     /* local size */
    int starts[2]   = {0,0};                        /* where this one starts */
    MPI_Datatype type, subarrtype;
    MPI_Type_create_subarray(2, sizes, subsizes, starts, MPI_ORDER_C, MPI_CHAR, &type);
    MPI_Type_create_resized(type, 0, gridsize/procgridsize*sizeof(char), &subarrtype);
    MPI_Type_commit(&subarrtype);

    char *globalptr=NULL;
    if (rank == 0) globalptr = &(global[0][0]);

    /* scatter the array to all processors */
    int sendcounts[procgridsize*procgridsize];
    int displs[procgridsize*procgridsize];

    if (rank == 0) {
        for (int i=0; i<procgridsize*procgridsize; i++) sendcounts[i] = 1;
        int disp = 0;
        for (int i=0; i<procgridsize; i++) {
            for (int j=0; j<procgridsize; j++) {
                displs[i*procgridsize+j] = disp;
                disp += 1;
            }
            disp += ((gridsize/procgridsize)-1)*procgridsize;
        }
    }


    MPI_Scatterv(globalptr, sendcounts, displs, subarrtype, &(local[0][0]),
                 gridsize*gridsize/(procgridsize*procgridsize), MPI_CHAR,
                 0, MPI_COMM_WORLD);

    /* now all processors print their local data: */

    for (int p=0; p<size; p++) {
        if (rank == p) {
            printf("Local process on rank %d is:\n", rank);
            for (int i=0; i<gridsize/procgridsize; i++) {
                putchar('|');
                for (int j=0; j<gridsize/procgridsize; j++) {
                    putchar(local[i][j]);
                }
                printf("|\n");
            }
        }
        MPI_Barrier(MPI_COMM_WORLD);
    }

    /* now each processor has its local array, and can process it */
    for (int i=0; i<gridsize/procgridsize; i++) {
        for (int j=0; j<gridsize/procgridsize; j++) {
            local[i][j] = 'A' + rank;
        }
    }

    /* it all goes back to process 0 */
    MPI_Gatherv(&(local[0][0]), gridsize*gridsize/(procgridsize*procgridsize),  MPI_CHAR,
                 globalptr, sendcounts, displs, subarrtype,
                 0, MPI_COMM_WORLD);

    /* don't need the local data anymore */
    free2dchar(&local);

    /* or the MPI data type */
    MPI_Type_free(&subarrtype);

    if (rank == 0) {
        printf("Processed grid:\n");
        for (int i=0; i<gridsize; i++) {
            for (int j=0; j<gridsize; j++) {
                putchar(global[i][j]);
            }
            printf("\n");
        }

        free2dchar(&global);
    }


    MPI_Finalize();

    return 0;
}

这篇关于使用MPI发送二维数组块用C的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！