[Mpi3-ft] General Network Channel Failure

[Josh Hursey]

It was mentioned in the conversation today that MPI_ERR_RANK_FAIL_STOP
may not be the first error returned by an MPI call. In particular the
MPI call may return an error symptomatic of a fail-stop process
failure (e.g., network link failed - currently MPI_ERR_OTHER), before
eventually diagnosing the event as a process failure. This 'space
between' MPI_SUCCESS behavior and MPI_ERR_RANK_FAIL_STOP behavior is
not currently defined, and probably should be for the application to
cleanly move from set of semantics for one error class to another.

The suggestion was to create a new general network error class (e.g.,
MPI_ERR_COMMUNICATION or MPI_ERR_NETWORK - MPI_ERR_COMM is taken) that
can be returned when the operation cannot complete due to network
issues (which might be later diagnosed as process failure and
escalated to the MPI_ERR_RANK_FAIL_STOP semantics). You could also
think about this error being used for network resource exhaustion as
well (something that Tony mentioned during the last MPI Forum
meeting). In which case retrying at a later time or taking some other
action before trying again would be useful/expected.

There are some issues with matching, and the implications on
collective operations. If the network error is sticky/permanent then
once the error is returned it will always be returned or escalated to
fail-stop process failure (or more generally to a 'higher/more
severe/more detailed' error class). A recovery proposal (similar to
what we are developing for process failure) would allow the
application to 'recover' the channel and continue communicating on it.


The feeling was that this should be expanded into a full proposal,
separate from the Run-Through Stabilization proposal. So we can
continue with the RTS proposal, and bring this forward when it is
ready.


What to folks think about this idea?

-- Josh

-- 
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://users.nccs.gov/~jjhursey