[Mpi3-ft] The state of MPI is undefined

[Josh Hursey]

On Mon, Jun 13, 2011 at 5:28 PM, Darius Buntinas <buntinas at mcs.anl.gov> wrote:
>
> On Jun 13, 2011, at 3:36 PM, Josh Hursey wrote:
>
>> On Mon, Jun 13, 2011 at 12:36 PM, Darius Buntinas <buntinas at mcs.anl.gov> wrote:
>>>
>>> I'm not sure I like the wording:
>>>
>>>  If the MPI implementation can continue operating after process failure
>>>  then it must return an appropriate error class (e.g.,
>>>  MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined
>>>  in Chapter 17.
>>>
>>> I don't think we can say that "if MPI returns error code X" ==> "MPI state is defined," because if some state corrupting error occurs, the state of MPI is undefined and the implementation might return error code X.  Rather, I think we should use the form "if error FOO occurs" ==> "MPI state will be BAR."  So for example, we can say "if an unrecoverable communication error occurs, the MPI implementation will return a MPI_ERR_COMMUNICATION for all pending communication operations and blah, blah, blah."
>>
>> For the case where the MPI was corrupted so that it is forced to
>> return a particular error code, I do not know how we can get around
>> the fact that we cannot define the state of MPI (the error forced the
>> MPI implementation to lie to the application). So even in the form "if
>> error FOO occurs" ==> "MPI state will be BAR" the 'will be' is too
>> strong since the corruption forced the return of FOO without the
>> consent of the MPI implementation.  But if we say 'might be' then we
>> are back to introducing a proposal that cannot be used by applications
>> - lack of solid semantics.
>
> Sorry, I should have been more specific.  When I say "if error FOO occurs"  I don't mean error FOO is returned but rather that an error condition occurred.  E.g., if a communication error occurs, an MPI_ERR_COMMUNICATION error will be returned."  So this explicitly excludes the state corrupting error.  Note that it's still possible that a state corrupting error results in MPI_ERR_COMMUNICATION to be returned, but we can't prevent that case.  But at least we can say that. e.g., assuming the state of the library is not already corrupted, for a compliant implementation, a communication error (however we define that) will not corrupt the library and the library will behave as prescribed.

Ah I see what you mean now. I think we should be aware of the corner
case and point it out to the users, but try not to have it diminish
the strength of the language of the standard as much as possible. Per
the advice to users below.

>
>>
>> So maybe we should add an advice to users that states that:
>> ------------
>> It is possible that the state of MPI becomes corrupted in an
>> undetectable manner. This undetectable error might force the MPI
>> implementation to return an error code (or even success)
>> unintentionally. For this pathological case there exist no mechanisms
>> to determine the correct running state. A high-quality implementation
>> will strive to return the correct return code from all MPI operations.
>> ------------
>
> I think this is OK, but maybe we can make it more general:
>
> It is possible that some error/fault* may cause the state of MPI to become corrupted in an undetectable manner.  The behavior of MPI in this case is undefined, and it is possible that the implementation returns incorrect error codes (or even MPI_SUCCESS).  So while a high-quality implementation will strive to always return correct return codes from MPI operations, it may not be possible in all cases.

I think that rewording of the advice to users is fine. Below is
slightly touched up:

--------------
It is possible that some error may cause the state of MPI to become
corrupted in an undetectable manner. The behavior of MPI in this case
is undefined, and it is possible that the implementation returns
incorrect error codes (including MPI_SUCCESS). So while a high-quality
implementation will strive to always return correct return codes from
MPI operations, it may not be possible in all cases.
--------------

>
>
> * Perhaps we should define error and fault and check for correct usage in the standard. Does anyone have good definitions?  Is an error a detected (at some level) fault?  Should we drop error and just use fault?

I have used the following before (there may be other, better
definitions out there):
---------------------
In a single system, an error is generated when a physical defect,
called a fault, is detected. A system failure is when the system
cannot deliver its intended function because of one or more errors
[A].
A fault-tolerant system will continue to operate normally in the
presence of errors.
[A] http://portal.acm.org/citation.cfm?id=1207694
---------------------

-- Josh

>
> -d
>
>>
>>>
>>> A blanket statement we can make about errors, then, would be something like "Unless otherwise noted in the standard, after an error is detected, the state of MPI is undefined."
>>>
>>> Another thought, we may get resistance from the Forum about the cost of checking for faults.  Maybe we discussed this already, I can't remember.  Do we want to make FT optional, and say "If the MPI implementation supports FT, then the behavior will be blah"?
>>
>> How about:
>> ------------------
>> MPI does not provide the user with transparent process recovery upon
>> process failure.
>> Once a process fails, MPI does not guarantee that the job can continue
>> or, if the job can continue, that the process can be recovered.
>> If a process failure occurs, and the MPI implementation is able to
>> correctly continue operating after process failure then the MPI
>> implementation will return an appropriate error class (e.g.,
>> MPI_ERR_RANK_FAIL_STOP) and provide the additional semantics defined
>> in Chapter 17.
>> The MPI implementation documentation will provide information on the
>> possible effect of each supported class of errors.
>>
>> Advice to users:
>> It is possible that the state of MPI becomes corrupted in an
>> undetectable manner. This undetectable error might force the MPI
>> implementation to return an error code (or even success)
>> unintentionally. For this pathological case there exist no mechanisms
>> to determine the correct running state. A high-quality implementation
>> will strive to return the correct return code from all MPI operations.
>> ------------------
>>
>> I like the idea of keeping the softer language with regard to whether
>> or not it is required, but instead say that if the user sees error
>> code X (in this case MPI_ERR_RANK_FAIL_STOP) then they can expect the
>> behavior outlined in Chapter 17. If they do not see that error, but
>> something else then it is implementation defined what will happen
>> afterwards. And the corner case is described to the users, which is an
>> understandable limitation of any software system.
>>
>>
>> What do folks think about this?
>>
>> -- Josh
>>
>>
>>>
>>> -d
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> mpi3-ft mailing list
>>> mpi3-ft at lists.mpi-forum.org
>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>>>
>>>
>>
>>
>>
>> --
>> Joshua Hursey
>> Postdoctoral Research Associate
>> Oak Ridge National Laboratory
>> http://users.nccs.gov/~jjhursey
>>
>> _______________________________________________
>> mpi3-ft mailing list
>> mpi3-ft at lists.mpi-forum.org
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>
>
> _______________________________________________
> mpi3-ft mailing list
> mpi3-ft at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>
>



-- 
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://users.nccs.gov/~jjhursey