[Mpi3-ft] Defining the state of MPI after an error
Richard Treumann
treumann at us.ibm.com
Thu Sep 30 09:52:05 CDT 2010
Josh & the FT Team
Thanks for this decision to suspend this ticket for now.
I am not in any position to judge whether the idea can be part of an
integrated approach to Fault Tolerance. If it, or something similar,
comes up in the context of an FT chapter and is well defined in that
context, I would not expect to have any objection.
I expect the discussion of any FT chapter brought to the Forum as a whole
to be difficult but I hope we are generally able so see it as a whole that
can be accepted as ready for a standards document.
If not that, I hope it can be recognized as broadly promising but not
mature enough to standardize. and moved to the Journal of R&D as a
proposal that people can implement and refine before a later decision to
standardize.
Dick
Dick Treumann - MPI Team
IBM Systems & Technology Group
Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601
Tele (845) 433-7846 Fax (845) 433-8363
From:
Joshua Hursey <jjhursey at open-mpi.org>
To:
"MPI 3.0 Fault Tolerance and Dynamic Process Control working Group"
<mpi3-ft at lists.mpi-forum.org>
Date:
09/29/2010 01:30 PM
Subject:
Re: [Mpi3-ft] Defining the state of MPI after an error
Sent by:
mpi3-ft-bounces at lists.mpi-forum.org
It was the feeling of the group during the teleconf that this proposal
should be suspended for the time being. I drafted a 'proposal resolution'
rationale distilled from all of the discussion so far and included it with
the proposed text on the wiki. If you feel that something is missing or
misrepresented in the resolution text, please let me know. This resolution
is this text that we will point the supporters of this idea to when they
ask about its status.
The text can be found at the following link:
https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue#ProposalResolution
-- Josh
On Sep 23, 2010, at 3:54 PM, Joshua Hursey wrote:
>
> On Sep 23, 2010, at 10:40 AM, Richard Treumann wrote:
>
>>
>> A few quick observations:
>>
>> 0)
>> The constant is MPI_ERROR_ARE_FATAL, not MPI_ERRORS_ABORT
>
> Fixed. Thanks
>
>>
>> 1)
>> The MPI standard only mandates one return code, MPI_SUCCESS. All other
return codes are implementation specific and non-portable. For
portability, MPI documents error classes and a query function that is
passed an implementation defined return code and returns the class.
>>
>> Assume I allow tags between 0 and 2**15. As an MPI implementor, I am
free to use return code 215 for a negative tag and 399 for one that is
above 2**15. The error message I print for 215 and the error message I
print for 399 can be different. If the user calls MPI_ERROR_CLASS() with
either 215 or 399 I give back the class MPI_ARR_TAG. The user who checks
the RC of a call to see if it is == MPI_ERR_TAG has written non-portable
code.
>>
>> If I decide return codes 251 and 399 must be in class
MPI_CANNOT_CONTINUE they can no longer be in class MPI_ERR_TAG.
>
> I understand, that is why MPI_ERR_CANNOT_CONTINUE is an error class. We
are not defining a new error code, but a new error class.
>
>>
>>
>> 2)
>> The MPI standard avoids mandating specific error checks. It identifies
a lot of errors and in many cases, says what error class that error is in.
It does not say an implementation MUST detect the error. I would not
violate the standard by skipping the check of whether MPI is initialized.
My customers may demand it but the standard does not. You are introducing
a mandate for one specific sort of error.
>
> Yes that is covered in the opening paragraph of 8.3. We are not
mandating that an implementation must detect errors, but that if it does
detect an error it must return an error code in an error class. So no
change in this regard from the existing standard. This proposal adds that
if the MPI implementation decides that it cannot continue after it detects
the original error then it must return an error code in the
MPI_ERR_CANNOT_CONTINUE error class on subsequent MPI calls. So I guess
you could say that it must detect that it is itself in an self-defined
erroneous state in order to return MPI_ERR_CANNOT_CONTINUE.
>
> On a related note, the standard language does not specify how an
implementation goes about implementing the MPI_ERR_CANNOT_CONTINUE
condition. The prototype will likely check this value upon entry to every
function call. A snazzy implementation may replace all the MPI function
calls by function pointers to a dummy function that only returns
MPI_ERR_CANNOT_CONTINUE. Resetting these function calls may occur in the
error handler at the time original error. This technique would not require
a check each time one enters the library. One could imagine other
techniques that would equally avoid the overhead of checking this global
variable.
>
>>
>> 3)
>> I am convinced that the intent of the standard is to require
MPI_ERROR_CLASS, MPI_ERROR_STRING and MPI_ABORT to work after an
ERRORS_RETURN. If this is insufficiently clear, it should probably be
addressed in a stand alone ticket. (it is certainly possible for an error
(detected or not) to trash internal state and for that to make one of
these three unusable but that applies to every MPI call. The standard does
not say MPI_Send must work even if state was scrambled by a wild store). I
do not know if anybody assumed MPI_INITIALIZED and MPI_FINALIZED must work
after an error. I see no harm in requiring it.
>
> It is insufficiently clear (to me at at least) from the current standard
if these functions are able to be used after an error. During the plenary
session last week, it was mentioned by a number of people that this
clarification should be part of the formal proposal for this new error
class. We can probably pull it out into a separate ticket, but they need
to be clarified as part of this ticket anyway.
>
>>
>> 5)
>> Finally - I do not see that the ticket does anything useful. In
particular, it does not provide any portability improvements I can see.
>>
>> The MPI implementation could offer a TIMID vs ADVENTUROUS switch
(environment variable)
>>
>> TIMID - MPI query functions like MPI_COMM_SIZE and MPI_ALLOC_MEM do not
trigger CANNOT_CONTINUE but every other error does.
>>
>> ADVENTUROUS - no error triggers CANNOT_CONTINUE.
>>
>> The default would probably need to be TIMID because if the default were
ADVENTUROUS, it would open the implementor to an accusation of failing to
protect the customer. There can be no such accusation now because the
standard does not imply the implementation should protect the customer.
>>
>> I have no clue from the ticket what would be a reasonable or portable
middle ground. I see the proposal as harmful because any attempt to use
it will produce an illusion of portability when implementors try to find a
middle ground without guidance form the standard.
>
> This thread has belabored the point that it is difficult to standardize
the middle ground. The expected behavior after an error is returned will
likely depend on the error class, MPI call, usage scenario, and, likely,
how the call is implemented. For this reason it is difficult to
standardize the behavior after any specific error class. If an
implementation finds some scenarios where it wishes to be adventurous and
continue working then it is allowed to do so, as long as it documents this
behavior.
>
> But what should the MPI implementation do if it finds that it cannot
continue working properly any more? It should probably return some type of
error to the user letting them know that it has stopped functioning. The
current standard says nothing about what should be returned in this
scenario. This proposal adds the MPI_ERR_CANNOT_CONTINUE error class to
fill this gap.
>
> With this new error class the user can now know that they should expect
MPI_ERR_CANNOT_CONTINUE from all future calls unless the MPI library is
attempting to recover. So if they detect a different error class or
MPI_SUCCESS then they know the implementation is up to something and to
proceed carefully in an implementation specific manner. With the current
standard, one implementation may use MPI_ERR_OTHER as the return class for
future calls where another implementation may use MPI_ERR_UNKNOWN and
still another MPI_ERR_INTERN. So an application is unable to determine if
the MPI_ERR_{OTHER|UNKNOWN|INTERN} was because of their use of the
interface, or because of the library ceasing to function properly. Instead
of mandating that one of the existing error classes fill this role, we
introduced the new error class. Now the application can use this as a
standardized red flag marking the limits of the ability of the MPI library
to function correctly.
>
>
> I acknowledge that this is a small advancement of the standard, but it
provides at least some bound on the undefined behavior. So now there is a
middle ground to speak of, instead of just everything else after the
error.
>
> -- Josh
>
>
>>
>> Dick Treumann - MPI Team
>> IBM Systems & Technology Group
>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601
>> Tele (845) 433-7846 Fax (845) 433-8363
>>
>>
>>
>> From: Joshua Hursey <jjhursey at open-mpi.org>
>> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working
Group" <mpi3-ft at lists.mpi-forum.org>
>> Date: 09/23/2010 08:57 AM
>> Subject: Re: [Mpi3-ft] Defining the state of MPI after an error
>> Sent by: mpi3-ft-bounces at lists.mpi-forum.org
>>
>> ________________________________
>>
>>
>>
>> (Bringing a lot of points together in a single response)
>>
>> The ticket that we are discussing is linked below (also part of the
very first email in this thread):
>>
https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/err_cannot_continue
>>
>> < snip >
>>
>> I deleted the discussion because only the ticket counts now.
>>
>>
>> <ATT00001..txt>
>
> ------------------------------------
> Joshua Hursey
> Postdoctoral Research Associate
> Oak Ridge National Laboratory
> http://www.cs.indiana.edu/~jjhursey
>
>
> _______________________________________________
> mpi3-ft mailing list
> mpi3-ft at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>
------------------------------------
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://www.cs.indiana.edu/~jjhursey
_______________________________________________
mpi3-ft mailing list
mpi3-ft at lists.mpi-forum.org
http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpi-forum.org/pipermail/mpiwg-ft/attachments/20100930/938b94f7/attachment-0001.html>
More information about the mpiwg-ft
mailing list