[Mpi3-ft] Persistent Communication & Sendrecv

[Joshua Hursey]

On Aug 30, 2010, at 5:35 PM, Joshua Hursey wrote:

> I was reviewing the point-to-point chapter today looking for wording/interfaces that need to be addressed in the FT proposal. I came across two sets of interfaces that probably deserve to be brought up on the list.
> 
> Examples and collective stuff coming soon :)
> 
> Persistent Communication:
> -------------------------
> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilization#a3.9PersistentCommunicationRequests
> 
> None of the persistent calls have a 'status' object associated with them, so I had to add some wording about failures during MPI_{*Send|Recv}_init and MPI_Start. The interesting case is if MPI_Startall() fails. What should it return (MPI_ERR_IN_STATUS)? How does an application determine which of the requests:
> - Started
> - Failed
> - Were not started
> 
> I think I devised a strategy for the application to check which started and which did not. But I wanted a few more eyes on this to see if there is a better way to deal with this situation.

I was thinking more about MPI_Startall() this morning and found a situation where this technique would not work.

If the application does:
--------------------
MPI_Send_init(rank=1, tag=123, req[0]);
MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, req[1]);
MPI_Send_init(rank=2, tag=123, req[2]);

MPI_Startall(3, req) // Fails with MPI_ERR_IN_STATUS
if( failed ) {
  for(i=0; i<3; ++i) {
    MPI_Request_get_status(req[i], flag, status);
    if( flag && status.error != success ) // Failed
    if( flag && status == empty ) // Not started
    if( flag && status != empty ) // Complete
  }
}
--------------------

The problem is with the definition of an 'empty' status which has (section 3.7.3):
---------------------
 source = MPI_ANY_SOURCE
 tag = MPI_ANY_TAG
 error = MPI_SUCCESS
 MPI_Get_count = 0
 MPI_test_cancelled = false
---------------------
So the successful completion of the MPI_Recv_init() call would be indistinguishable from the 'not started' or inactive state of the call.

I see three options (without changing the function interface):
1) Add a field to the MPI_Status object to determine if the request is active or not. MPI_Status.MPI_IS_ACTIVE returns false if inactive, true if active.

2) Change the definition of an 'empty' status to allow the user to distinguish it from the completed MPI_Recv_init request above. (Backwards compatibility issues here).

3) Since MPI_Start and MPI_Startall do not specify what happens when an active request is passed to them (one that has been started, but not completed) we could add some language that works in our benefit.

> If an active request is passed to MPI_Start{all} the function will immediately return MPI_ERR_PENDING without starting any of the requests.

After an error from MPI_Startall the user could use MPI_Start to distinguish the successful completion from the not started. The problem being that it would automatically start the request that was previously not started. Is this a problem?

So the above code would be changed to:
--------------------
MPI_Send_init(rank=1, tag=123, req[0]);
MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, req[1]);
MPI_Send_init(rank=2, tag=123, req[2]);

MPI_Startall(3, req) // Fails with MPI_ERR_IN_STATUS
if( failed ) {
  for(i=0; i<3; ++i) {
    MPI_Request_get_status(req[i], flag, status);
    if( flag && status.error != success ) // Failed
    if( flag ) {
      ret = MPI_Start(req[i])
      if( ret == MPI_ERR_PENDING ) // Complete
      if( ret == MPI_SUCCESS ) // Previously not started, now active
    }
  }
}
--------------------

Thoughts? Preferences?

-- Josh

> 
> 
> Sendrecv{_replace}
> ------------------
> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilization#a3.10Send-Receive
> 
> These interfaces only have one status object associated with them. So if both ranks fail, only one of them can be indicated in the status object. So I wrote up some text to add to this section for the application to determine what happened. Another set of eyes here would be appreciated just to make sure I am not missing anything.
> 
> 
> Thanks,
> Josh
> 
> ------------------------------------
> Joshua Hursey
> Postdoctoral Research Associate
> Oak Ridge National Laboratory
> http://www.cs.indiana.edu/~jjhursey
> 
> 
> 
> 
> 
> _______________________________________________
> mpi3-ft mailing list
> mpi3-ft at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
> 

------------------------------------
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://www.cs.indiana.edu/~jjhursey