[Mpi3-ft] MPI_ANY_SOURCE

[Josh Hursey]

Currently we state:
----------------------
17.6 Point-to-Point Communication
...
When a process detects a new process failure, the ability to perform
wildcard receives (i.e., receives where MPI_ANY_SOURCE has been
specified for the source parameter) will be disabled on all
communicators that contain the failed process. When wildcard receives
are disabled on a communicator, all pending wildcard receive
operations on that communicator are completed and an error with class
MPI_ERR_PROC_FAIL_STOP will be returned for those operations. Any new
wildcard receive operations posted to a communicator with disabled
wildcard receives will be immediately completed and return an error
code of the class MPI_ERR_PROC_FAIL_STOP.
Wildcard receives can be re-enabled with the
MPI_COMM_REENABLE_ANY_SOURCE function described below.
----------------------

The problem is that by completing the pending wildcard receives with
an error, we may cause unintended matching of concurrent receives. For
example:
-------------------
Proc 0                Proc 1
Irecv(ANY,TAGX); [A]
Irecv(1, TAGX); [B]
Waitall()
/******* Proc 2 fails *********/
                      Send(0, TAGX) [C]
                      Send(0, TAGX) [D]
-------------------
The intention is that Irecv[A] will match Send[C] and Irecv[B] will
match Send[D]. But if another process fails (proc 2 in this example),
then Irecv[A] will complete in error. Then Irecv[B] will match
Send[C], and Send[D] will remain unmatched.


We need a mechanism that provides notification to the application
waiting on an ANY_SOURCE receive that some process failed in this
communicator while still providing the necessary matching guarantees.
The user can then choose if they want to cancel the operation or
continue waiting.


On the teleconference today, we were thinking through a warning based
concept. Instead of completing the pending nonblocking ANY_SOURCE
receives, we would return a special warning code (say
MPI_WARN_PROC_FAIL_STOP). The message would -not- be completed, but
return to the user to decide if they wish to continue waiting or
cancel the offending receive operation.

For blocking ANY_SOURCE receives, the user would be returned an error
(MPI_ERR_PROC_FAIL_STOP), and the operation would be completed in
error. Since we do not have a request handle, there is no way to keep
this receive active while the situation is resolved. So we complete
the receive with an error, in a sense canceling the receive operation.

For nonblocking ANY_SOURCE receives, the user would be returned a
warning (MPI_WARN_PROC_FAIL_STOP), and the request would remain
active. The application would then be able to either re-enable
ANY_SOURCE (via the API call) or cancel the offending receive
operation. The user is in control over the matching guarantees since
the request remains active, but cannot complete without action from
the user.


In the example above, Proc 0 would return MPI_ERR_IN_STATUS from
MPI_WAITALL. The status of the active requests would be
MPI_ERR_PENDING (per section 3.7.3), except for the offending
ANY_SOURCE operations which will have the error field set to
MPI_WARN_PROC_FAIL_STOP. So Irecv[A] would be MPI_ERR_PENDING, and
Irecv[B] would be MPI_WARN_PROC_FAIL_STOP. The user can then associate
the error with a specific request. Once a request is identified the
user can either cancel it, or re-enable ANY_SOURCE on the communicator
and continue waiting.


So I think this fixes the matching issue with the interface, but I'm
not sure if it addresses the concerns of the hardware matching folks.
Additionally, this proposal adds a new state to the request concept
'active but not completable without action by the user' and we need to
think if this requires semantic qualification elsewhere in the
standard.


What to folks think about this solution?

-- Josh

-- 
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://users.nccs.gov/~jjhursey