[Mpi3-ft] Defining the state of MPI after an error

[Joshua Hursey]

On Sep 23, 2010, at 10:40 AM, Richard Treumann wrote:

> 
> A few quick observations:
> 
> 0)
> The constant is MPI_ERROR_ARE_FATAL, not MPI_ERRORS_ABORT

Fixed. Thanks

> 
> 1)
> The MPI standard only mandates one return code, MPI_SUCCESS. All other return codes are implementation specific and non-portable.  For portability, MPI documents  error classes and a query function that is passed an implementation defined return code and returns the class.
> 
> Assume I allow tags between 0 and 2**15. As an MPI implementor, I am free to use return code 215 for a negative tag and 399 for one that is above 2**15.  The error message I print for 215 and the error message I print for 399 can be different. If the user calls MPI_ERROR_CLASS() with either 215  or 399 I give back the class MPI_ARR_TAG.  The user who checks the RC of a call to see if it is == MPI_ERR_TAG has written non-portable code.
> 
> If I decide  return codes 251 and 399 must be in class MPI_CANNOT_CONTINUE they can no longer be in class MPI_ERR_TAG.

I understand, that is why MPI_ERR_CANNOT_CONTINUE is an error class. We are not defining a new error code, but a new error class.

> 
> 
> 2)
> The MPI standard avoids mandating specific error checks.  It identifies a lot of errors and in many cases, says what error class that error is in.  It does not say an implementation MUST detect the error. I would not violate the standard by skipping the check of whether MPI is initialized. My customers may demand it but the standard does not.  You are introducing a mandate for one specific sort of error.

Yes that is covered in the opening paragraph of 8.3. We are not mandating that an implementation must detect errors, but that if it does detect an error it must return an error code in an error class. So no change in this regard from the existing standard. This proposal adds that if the MPI implementation decides that it cannot continue after it detects the original error then it must return an error code in the MPI_ERR_CANNOT_CONTINUE error class on subsequent MPI calls. So I guess you could say that it must detect that it is itself in an self-defined erroneous state in order to return MPI_ERR_CANNOT_CONTINUE.

On a related note, the standard language does not specify how an implementation goes about implementing the MPI_ERR_CANNOT_CONTINUE condition. The  prototype will likely check this value upon entry to every function call. A snazzy implementation may replace all the MPI function calls by function pointers to a dummy function that only returns MPI_ERR_CANNOT_CONTINUE. Resetting these function calls may occur in the error handler at the time original error. This technique would not require a check each time one enters the library. One could imagine other techniques that would equally avoid the overhead of checking this global variable.

> 
> 3)
> I am convinced that the intent of the standard is to require MPI_ERROR_CLASS, MPI_ERROR_STRING and MPI_ABORT to work after an ERRORS_RETURN. If this is insufficiently clear, it should probably be addressed in a stand alone ticket.  (it is certainly possible for an error (detected or not) to trash internal state and for that to make one of these three unusable but that applies to every MPI call. The standard does not say MPI_Send must work even if state was scrambled by a wild store).  I do not know if anybody assumed MPI_INITIALIZED and MPI_FINALIZED must work after an error. I see no harm in requiring it.

It is insufficiently clear (to me at at least) from the current standard if these functions are able to be used after an error. During the plenary session last week, it was mentioned by a number of people that this clarification should be part of the formal proposal for this new error class. We can probably pull it out into a separate ticket, but they need to be clarified as part of this ticket anyway.

> 
> 5)
> Finally - I do not see that the ticket does anything useful.  In particular, it does not provide any portability improvements I can see.
> 
> The MPI implementation could offer a TIMID vs ADVENTUROUS switch (environment variable)
> 
> TIMID - MPI query functions like MPI_COMM_SIZE and MPI_ALLOC_MEM do not trigger CANNOT_CONTINUE but every other error does.
> 
> ADVENTUROUS - no error triggers CANNOT_CONTINUE.
> 
> The default would probably need to be TIMID because if the default were ADVENTUROUS, it would open the implementor to an accusation of failing to protect the customer. There can be no such accusation now because the standard does not imply the implementation should protect the customer.
> 
> I have no clue from the ticket what would be a reasonable or portable middle ground.    I see the proposal as harmful because any attempt to use it will produce an illusion of portability when implementors try to find a middle ground without guidance form the standard.

This thread has belabored the point that it is difficult to standardize the middle ground. The expected behavior after an error is returned will likely depend on the error class, MPI call, usage scenario, and, likely, how the call is implemented. For this reason it is difficult to standardize the behavior after any specific error class. If an implementation finds some scenarios where it wishes to be adventurous and continue working then it is allowed to do so, as long as it documents this behavior.

But what should the MPI implementation do if it finds that it cannot continue working properly any more? It should probably return some type of error to the user letting them know that it has stopped functioning. The current standard says nothing about what should be returned in this scenario. This proposal adds the MPI_ERR_CANNOT_CONTINUE error class to fill this gap.

With this new error class the user can now know that they should expect MPI_ERR_CANNOT_CONTINUE from all future calls unless the MPI library is attempting to recover. So if they detect a different error class or MPI_SUCCESS then they know the implementation is up to something and to proceed carefully in an implementation specific manner. With the current standard, one implementation may use MPI_ERR_OTHER as the return class for future calls where another implementation may use MPI_ERR_UNKNOWN and still another MPI_ERR_INTERN. So an application is unable to determine if the MPI_ERR_{OTHER|UNKNOWN|INTERN} was because of their use of the interface, or because of the library ceasing to function properly. Instead of mandating that one of the existing error classes fill this role, we introduced the new error class. Now the application can use this as a standardized red flag marking the limits of the ability of the MPI library to function correctly.


I acknowledge that this is a small advancement of the standard, but it provides at least some bound on the undefined behavior. So now there is a middle ground to speak of, instead of just everything else after the error.

-- Josh


> 
> Dick Treumann  -  MPI Team
> IBM Systems & Technology Group
> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601
> Tele (845) 433-7846         Fax (845) 433-8363
> 
> 
> 
> From:   Joshua Hursey <jjhursey at open-mpi.org>
> To:     "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft at lists.mpi-forum.org>
> Date:   09/23/2010 08:57 AM
> Subject:        Re: [Mpi3-ft] Defining the state of MPI after an error
> Sent by:        mpi3-ft-bounces at lists.mpi-forum.org
> 
> ________________________________
> 
> 
> 
> (Bringing a lot of points together in a single response)
> 
> The ticket that we are discussing is linked below (also part of the very first email in this thread):
> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue
> 
> < snip >
> 
> I deleted the discussion because only the ticket counts now.
> 
> 
> <ATT00001..txt>

------------------------------------
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://www.cs.indiana.edu/~jjhursey