[Mpi3-rma] Request-Based RMA Example

[James Dinan]

Hi All,

Example is below, please give any feedback.  I'll work on compressing 
this down the match the format in the writeup.

Caption: The following example shows how request-based operations can be 
used to overlap communication with computation.  Each process fetches, 
processes, and writes the result for $NSTEPS$ chunks of data.  Instead 
of a single buffer, $M$ local buffers are used to allow up to $M$ 
communication operations to overlap with computation.

  ~Jim.

---snip---

int i, j;
MPI_Window  win;
MPI_Request put_req[M] = { MPI_REQUEST_NULL };
MPI_Request get_req;
double      data[M][N];

/* Create win: size M*N*sizeof(double), displacement unit MPI_DOUBLE */

MPI_Win_lock_all(0, win);

for (i = 0; i < NSTEPS; i++) {
   MPI_Waitany(M, put_req, &j, MPI_STATUS_IGNORE);
   MPI_RGet(data[j], N, MPI_DOUBLE, target, i*N, N, MPI_DOUBLE, win, 
&get_req);
   MPI_Wait(get_req);
   compute(i, data[M], ...);
   MPI_RPut(data[j], N, MPI_DOUBLE, target, i*N, N, MPI_DOUBLE, win, &R[j]);
}

MPI_Waitall(M, put_req, MPI_STATUSES_IGNORE);
MPI_Win_unlock_all(win);