[Mpi3-ft] The state of MPI is undefined

[Josh Hursey]

On Mon, Jun 13, 2011 at 12:36 PM, Darius Buntinas <buntinas at mcs.anl.gov> wrote:
>
> I'm not sure I like the wording:
>
>  If the MPI implementation can continue operating after process failure
>  then it must return an appropriate error class (e.g.,
>  MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined
>  in Chapter 17.
>
> I don't think we can say that "if MPI returns error code X" ==> "MPI state is defined," because if some state corrupting error occurs, the state of MPI is undefined and the implementation might return error code X.  Rather, I think we should use the form "if error FOO occurs" ==> "MPI state will be BAR."  So for example, we can say "if an unrecoverable communication error occurs, the MPI implementation will return a MPI_ERR_COMMUNICATION for all pending communication operations and blah, blah, blah."

For the case where the MPI was corrupted so that it is forced to
return a particular error code, I do not know how we can get around
the fact that we cannot define the state of MPI (the error forced the
MPI implementation to lie to the application). So even in the form "if
error FOO occurs" ==> "MPI state will be BAR" the 'will be' is too
strong since the corruption forced the return of FOO without the
consent of the MPI implementation.  But if we say 'might be' then we
are back to introducing a proposal that cannot be used by applications
- lack of solid semantics.

So maybe we should add an advice to users that states that:
------------
It is possible that the state of MPI becomes corrupted in an
undetectable manner. This undetectable error might force the MPI
implementation to return an error code (or even success)
unintentionally. For this pathological case there exist no mechanisms
to determine the correct running state. A high-quality implementation
will strive to return the correct return code from all MPI operations.
------------

>
> A blanket statement we can make about errors, then, would be something like "Unless otherwise noted in the standard, after an error is detected, the state of MPI is undefined."
>
> Another thought, we may get resistance from the Forum about the cost of checking for faults.  Maybe we discussed this already, I can't remember.  Do we want to make FT optional, and say "If the MPI implementation supports FT, then the behavior will be blah"?

How about:
------------------
MPI does not provide the user with transparent process recovery upon
process failure.
Once a process fails, MPI does not guarantee that the job can continue
or, if the job can continue, that the process can be recovered.
If a process failure occurs, and the MPI implementation is able to
correctly continue operating after process failure then the MPI
implementation will return an appropriate error class (e.g.,
MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined
in Chapter 17.
The MPI implementation documentation will provide information on the
possible effect of each supported class of errors.

Advice to users:
It is possible that the state of MPI becomes corrupted in an
undetectable manner. This undetectable error might force the MPI
implementation to return an error code (or even success)
unintentionally. For this pathological case there exist no mechanisms
to determine the correct running state. A high-quality implementation
will strive to return the correct return code from all MPI operations.
------------------

I like the idea of keeping the softer language with regard to whether
or not it is required, but instead say that if the user sees error
code X (in this case MPI_ERR_RANK_FAIL_STOP) then they can expect the
behavior outlined in Chapter 17. If they do not see that error, but
something else then it is implementation defined what will happen
afterwards. And the corner case is described to the users, which is an
understandable limitation of any software system.


What do folks think about this?

-- Josh


>
> -d
>
>
>
>
> _______________________________________________
> mpi3-ft mailing list
> mpi3-ft at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>
>



-- 
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://users.nccs.gov/~jjhursey