<br><font size=2 face="sans-serif">The current MPI standard does not say
the MPI implementation is totally broken once there is an error. Saying
MPI state is undefined after an error simply says that the detailed semantic
of the MPI standard can no longer be promised. In other words, after an
error you leave behind the security of a portable standard semantic. You
are operating at your own risk. You do not need to read more than that
into it.</font>
<br>
<br><font size=2 face="sans-serif">IBM MPI does not make any attempt to
prevent future calls into libmpi from working. If they work in the way
you expect, cool. If you want to use MPI communication in a tool
that runs on its own threads and helps with failure analysis. go for it.
If you want to keep using MPI to design an airplane wing, please warn me.
I do not want to be on that plane. MPI may look like it is still working
perfectly but not be. For a tool, it is no big deal but for an airplane
wing design, it is.</font>
<br>
<br><font size=2 face="sans-serif">I think having an MPI implementation
return </font><tt><font size=2>MPI_ERR_CANNOT_CONTINUE</font></tt><font size=2 face="sans-serif">
for every MPI call after the error is way beyond what the standard intends.
The use case Bronis offered would be totally broken by this idea.</font>
<br>
<br><font size=2 face="sans-serif">In IBM MPI, if you set MPI_ERRORS_RETURN
and call MPI_Allreduce on MPI_COMM_WORLD but have a bad MPI_Op on
some, the ranks with the bad OP will return and the ones with no error
will hang forever. You will never break the hung threads free. The
threads that returned (on the ranks with an error) can call MPI_Abort
and kill the job. </font>
<br>
<br><font size=2 face="sans-serif">On the other hand, if you already have
a dup of MPI_COMM_WORLD called MPI_tool_world and a tool thread on each
task, you can communicate among tool threads using MPI_tool_world and I
would expect that to work fine. As far as I know, everything in the MPI
standard that does not depend on MPI_COMM_WORLD will still work. An
attempt to do a collective call on MPI_COMM_WORLD may go wrong.</font>
<br>
<br><font size=2 face="sans-serif">Does that make passing a bad MPI_Op
to MPI_Allreduce a CAN or CANNOT CONTINUE? How would I decide?</font>
<br>
<br>
<br><font size=2 face="sans-serif">BTW - If new functions that may be unsuccessful
without damage to MPI state are added to the standard they use a TRY semantic.
Just avoid using the return code to report that the try did not succeed.
The simplest example would be:</font>
<br>
<br><font size=2 face="sans-serif">MPI_TRY_ALLOC_MEM(size, info, baseptr,
flag)</font>
<br>
<br><font size=2 face="sans-serif">If size is a negative number, this could
return an error in class MPI_ERR_ARG but it would never return an error
for a memory shortage. The flag would tell the user if the memory had been
allocated. A flag of FALSE and an RC of MPI_SUCCESS allows MPI to stay
in a defined state and the user to try again with a smaller memory request.</font>
<br>
<br><font size=2 face="sans-serif"> Dick
</font>
<br>
<br>
<br>
<br>
<br>
<br>
<br><font size=2 face="sans-serif">Dick Treumann - MPI Team
<br>
IBM Systems & Technology Group<br>
Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601<br>
Tele (845) 433-7846 Fax (845) 433-8363<br>
</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">From:</font>
<td><font size=1 face="sans-serif">Joshua Hursey <jjhursey@open-mpi.org></font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">To:</font>
<td><font size=1 face="sans-serif">"MPI 3.0 Fault Tolerance and Dynamic
Process Control working Group" <mpi3-ft@lists.mpi-forum.org></font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Date:</font>
<td><font size=1 face="sans-serif">09/21/2010 12:46 PM</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Subject:</font>
<td><font size=1 face="sans-serif">Re: [Mpi3-ft] Defining the state of
MPI after an error</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Sent by:</font>
<td><font size=1 face="sans-serif">mpi3-ft-bounces@lists.mpi-forum.org</font></table>
<br>
<hr noshade>
<br>
<br>
<br><tt><font size=2>The MPI_ERR_CANNOT_CONTINUE proposal in a small step
in the direction to a more well defined set of expected/correct behavior
after an error. Currently no standard MPI call can describe expected behavior
after a failure since there is an overriding statement about the state
being undefined. The CANNOT_CONTINUE error class defines a limit on this
behavior, while still allowing the MPI implementation to decide what is
correct behavior after the error for their implementation. This proposal
seems small, almost inconsequential, but it opens the door a bit more for
exploring well defined error semantics in future proposals.<br>
<br>
If an application wishes to continue using the MPI implementation after
an error, it is still at their own risk. But now there is a well defined
error class for the MPI implementation to hand back to the application
when it does not know how to correctly proceed with this function call.
The hope is that this proposal encourages implementations to start defining
reasonable error semantics for specific error scenarios. These emerging
error semantics are then able to be brought forward for eventual standardization.<br>
<br>
So for the Bcast example that you cited, I don't think I can say exactly
what expected behavior should be. It is an interesting case, since even
in the current standard it does not say anything about what the user can
expect to happen if only one of the processes receive a local error. Does
the collective fail? Does it hang? Most implementations probably just hang
the other dependent processes in the collective, since doing more could
hinder performance. So the user will have to have another mechanism to
interrupt the collective operation (maybe a separate checking thread, or
maybe relying on a tool to help detect such scenarios). This proposal just
says that it is fine for the MPI library to block the erroneous process
from making future calls into MPI. If the MPI implementation allows for
something more reasonable to happen, it is allowed to do so. If an MPI
implementation does so, and users like the semantics other implementations
may be pushed to support the same semanti!<br>
cs, and now we are on the road to standardization.<br>
<br>
So when someone brings forward a new proposal for an new interface (like
the Shared Memory allocation routines), and they have a few error class
scenarios that are informative or warning in nature they can specify what
expected behavior should be for continued operation from those specific
error classes. Without this proposal, defining such expected behavior in
the standard is ill advised since there is the blanket statement of undefined
behavior. With the proposal, it then becomes appropriate to include these
semantics in the standard since the undefined behavior is able to be bound
by the MPI_ERR_CANNOT_CONTINUE error class.<br>
<br>
So a standard compliant basic MPI implementation will always return MPI_ERR_CANNOT_CONTINUE
after an error on subsequent calls. If a new interface is included in a
future standard that has well defined semantics on the behavior of MPI
after, say, the MPI_ERR_RANK error class. Then this basic MPI implementation
will return MPI_ERR_CANNOT_CONTINUE after an error on subsequent calls
except for this case in which it must behave according to the standard.<br>
<br>
As we are seeing with attempting to define sound semantics for the new
MPI_ERR_RANK_FAIL_STOP error class in the stabilization proposal, doing
so is a lot of work. So there is a challenge in attempting to define the
state of MPI (and further the distributed environment) after any error
class. But if it is important to applications it may be worth it.<br>
<br>
It is my opinion that the current standard's blanket language is too restrictive.
So I see the MPI_ERR_CANNOT_CONTINUE proposal as a small step towards better
error semantics. Though, as with anything, I can be convinced otherwise
if there is doubt about my reasoning or the utility of such an error class.<br>
<br>
Thanks for your patience in this thread.<br>
<br>
-- Josh<br>
<br>
On Sep 20, 2010, at 2:34 PM, Richard Treumann wrote:<br>
<br>
> <br>
> Josh<br>
> <br>
> Did you see this?<br>
> <br>
> "For example, if there were a loop of 100 MPI_Bcast calls and
on iteration 5, rank 3 uses a bad communicator, what is the proper state?
Either a sequence number is mandated so the other ranks hang quickly
or a sequence number is prohibited so everybody keeps going until the "end"
when the missing MPI_Bcast becomes critical. Of course, with no sequence
number, some tasks are stupidly using the iteration n-1 data for their
iteration n computation. "<br>
> <br>
> If my MPI implementation is going to tell you it "allows for
correct operation after returning that error class", the standard
needs to tell me which behavior is "correct operation" and what
is not "correct operation".<br>
> <br>
> As I mentioned to Bronis, I have no problem with an advise to implementors
urging that they at least allow an option for continued use of libmpi,
after an error returns, but at the users own risk. Offhand,
I cannot see any harm in urging MPI implementors to promise that all locks
will be released when a subroutine returns non-SUCCESS.<br>
> <br>
> Can you give me a use case for the added complexity of CAN_CONTINUE
vs CANNOT_CONTINUE.?<br>
> <br>
> Consider that an error class of CAN_CONTINUE that is returned because
the MPI implementor does not really know what "correct operation"
means and decides to wing it is not very useful. An MPI implementor
that can get in real trouble for negligence would need to be very cautious
about returning CAN_CONTINUE.<br>
> <br>
> Just asking the MPI implementor to refrain from flagging the error
"MPI disabled after prior error" on otherwise correct future
calls seems as good to me.<br>
> <br>
> <br>
> Dick Treumann - MPI Team<br>
> <br>
> <br>
> IBM Systems & Technology Group<br>
> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601<br>
> Tele (845) 433-7846 Fax (845) 433-8363<br>
> <br>
> <br>
> <br>
> From: Joshua Hursey <jjhursey@open-mpi.org><br>
> To: "MPI 3.0 Fault Tolerance and Dynamic Process
Control working Group" <mpi3-ft@lists.mpi-forum.org><br>
> Date: 09/20/2010 01:51 PM<br>
> Subject: Re: [Mpi3-ft] Defining the state
of MPI after an error<br>
> Sent by: mpi3-ft-bounces@lists.mpi-forum.org<br>
> <br>
> ________________________________<br>
> <br>
> <br>
> <br>
> So the proposal makes no requirements about the state of the distributed
environment after an error. All it defines is a error class to be returned
by an MPI implementation once the MPI implementation can no longer function
correctly. This provides a necessary feedback mechanism for the application
to determine if and how the MPI implementation can be used after an error
occurs. It is the responsibility of the application to avoid deadlocks
and other such issues that can result from handling and recovering from
errors. If an application is designed to recover from MPI_ERR_TAG that's
great. If the MPI library allows for correct operation after returning
that error class, then even better. If the MPI library cannot continue
operation after that error then it can block subsequent operations by returning
MPI_ERR_CANNOT_CONTINUE.<br>
> <br>
> I disagree with your assessment that this will be difficult to implement/test
since a trivial implementation of this proposal is to set a global variable
when an error occurs to always return MPI_ERR_CANNOT_CONTINUE when the
application calls into the MPI library (it is a similar check as the 'is_mpi_initialized'
check that has to be there anyway). If an implementation wants to do more
(and it is definitely not required to do so) then it can define that in
it's documentation.<br>
> <br>
> If an application wants to try to use MPI after an error it must understand
that the error is local in nature (it cannot assume that every process
received an error). If it can figure out how to recover from it, and the
MPI implementation is able to function correctly afterward then we should
let them figure it out. This allows us to define the boundaries of correct
operation after an error. Before the application -could- keep using the
MPI library after an error, but it was entirely undefined and not-portable
what would happen. Now the application can portably attempt to use the
MPI library after an error and know that it can expect either normal functionality
(for those few implementations that do more than the minimum necessary)
or MPI_ERR_CANNOT_CONTINUE in which the library locks them out and they
then know to terminate normally.<br>
> <br>
> I hope this helps a bit, but maybe I am missing the core problem that
you are trying to get at.<br>
> <br>
> -- Josh<br>
> <br>
> On Sep 20, 2010, at 1:09 PM, Bronis R. de Supinski wrote:<br>
> <br>
>> <br>
>> Dick:<br>
>> <br>
>> Re:<br>
>>> I did not intend to ignore your use case.<br>
>> <br>
>> No problem.<br>
>> <br>
>>> I did mention that I have no worries about asking MPI implementations<br>
>>> to refrain from blocking future MPI calls after an error is
detected.<br>
>>> That was an implicit recognition of your use case.<br>
>> <br>
>> OK, that helps.<br>
>> <br>
>>> The MPI standard already forbids having an MPI call on one
thread block<br>
>>> progress on other threads. I would interpret that to
include a case<br>
>>> where a thread is blocked in a collective communication or
a MPI_Recv<br>
>>> that will never be satisfied. That is, the blocked MPI call
cannot<br>
>>> prevent other threads from using libmpi. Requiring libmpi
to release<br>
>>> any lock it took even when doing an error return would be
logical but<br>
>>> may not be implied by what is currently written.<br>
>> <br>
>> The current text provides no such guarantee. Once anerror is<br>
>> returned anywhere, all bets are off (at least that is how I<br>
>> have read it; I would need to go back through the text to<br>
>> find the exact words that cause my concern).<br>
>> <br>
>>> Communicators provide a sort of isolation that keeps stray
crap from<br>
>>> failed operations from spilling over (such as eager sent message
for<br>
>>> which the MPI_Recv failed). If the tool uses its own
threads and<br>
>>> private communicators, I agree it is reasonable to ask any
libmpi to<br>
>>> avoid sabotaging that communication.<br>
>> <br>
>> That would be perfect from my perspective.<br>
>> <br>
>>> Where I get concerned is when we start talking about affirmative<br>
>>> requirements for distributed MPI state after an error<br>
>> <br>
>> I don't think we can have those beyond "best effort".<br>
>> The errors may indicate problems that make further<br>
>> communication impossible -- perhaps because of the<br>
>> erroneous action or just due to the state of the<br>
>> network or other processes. I do think we can require<br>
>> accurate return values and have an advice to implementers<br>
>> that suggests best effort following errors. I believe<br>
>> that would satisfy my requirements.<br>
>> <br>
>> Bronis<br>
>> <br>
>> <br>
>> <br>
>> <br>
>>> <br>
>>> Dick<br>
>>> <br>
>>> Dick Treumann - MPI Team<br>
>>> IBM Systems & Technology Group<br>
>>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY
12601<br>
>>> Tele (845) 433-7846 Fax (845)
433-8363<br>
>>> <br>
>>> <br>
>>> <br>
>>> From: "Bronis R. de Supinski" <bronis@llnl.gov><br>
>>> To: "MPI 3.0 Fault Tolerance and Dynamic
Process Control working Group" <mpi3-ft@lists.mpi-forum.org><br>
>>> Date: 09/20/2010 12:46 PM<br>
>>> Subject: Re: [Mpi3-ft] Defining
the state of MPI after an error<br>
>>> Sent by: mpi3-ft-bounces@lists.mpi-forum.org<br>
>>> <br>
>>> ________________________________<br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> Dick:<br>
>>> <br>
>>> You seem to be ignoring my use case. Specifically, I<br>
>>> have tool threads that use MPI. Their use of MPI should<br>
>>> be unaffected by all of the scenarios that you are raising.<br>
>>> However, the standard provides no way for me to tell if<br>
>>> they work correctly in these situations. I just have to<br>
>>> cross my fingers and hope.<br>
>>> <br>
>>> FYI: Your implementation has long met this requirement<br>
>>> (my hopes are not dashed with it). Others have begun to<br>
>>> recently. In any event, I would like some way to tell...<br>
>>> <br>
>>> Further, it is useful in many other scenarios apply to know<br>
>>> that the implementation intends to remain usable. I am not<br>
>>> looking for a promise of correct execution; I am looking<br>
>>> for a promise of best effort and accurate return codes.<br>
>>> <br>
>>> Bronis<br>
>>> <br>
>>> <br>
>>> <br>
>>> On Mon, 20 Sep 2010, Richard Treumann wrote:<br>
>>> <br>
>>>> <br>
>>>> If there is any question about whether these calls are
still valid after an error with an error handler that returns (MPI_ERRORS_RETURN
or user handler)<br>
>>>> <br>
>>>> MPI_Abort,<br>
>>>> MPI_Error_string<br>
>>>> MPI_Error_class<br>
>>>> <br>
>>>> I assume it should be corrected as a trivial oversight
in the original text.<br>
>>>> <br>
>>>> I would regard the real issue as being the difficulty
with assuring the state of remote processes.<br>
>>>> <br>
>>>> There is huge difficulty in making any promise about how
an interaction between a process that has not taken an error and one that
has will behave.<br>
>>>> <br>
>>>> For example, if there were a loop of 100 MPI_Bcast calls
and on iteration 5, rank 3 uses a bad communicator, what is the proper
state? Either a sequence number is mandated so the other ranks hang
quickly or a sequence number is prohibited so everybody keeps going until
the "end" when the missing MPI_Bcast becomes critical. Of
course, with no sequence number, some tasks are stupidly using the iteration
n-1 data for their iteration n computation.<br>
>>>> <br>
>>>> <br>
>>>> <br>
>>>> <br>
>>>> <br>
>>>> <br>
>>>> Dick Treumann - MPI Team<br>
>>>> IBM Systems & Technology Group<br>
>>>> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie,
NY 12601<br>
>>>> Tele (845) 433-7846 Fax (845)
433-8363<br>
>>>> <br>
>>> _______________________________________________<br>
>>> mpi3-ft mailing list<br>
>>> mpi3-ft@lists.mpi-forum.org<br>
>>> </font></tt><a href="http://blockedlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft"><tt><font size=2>http://BLOCKEDlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</font></tt></a><tt><font size=2><</font></tt><a href="http://blockedlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft"><tt><font size=2>http://blockedlists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</font></tt></a><tt><font size=2>><br>
>>> <br>
>>> <br>
>>> <br>
>> _______________________________________________<br>
>> mpi3-ft mailing list<br>
>> mpi3-ft@lists.mpi-forum.org<br>
>> </font></tt><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft"><tt><font size=2>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</font></tt></a><tt><font size=2><br>
>> <br>
> <br>
> ------------------------------------<br>
> Joshua Hursey<br>
> Postdoctoral Research Associate<br>
> Oak Ridge National Laboratory<br>
> </font></tt><a href=http://www.cs.indiana.edu/~jjhursey><tt><font size=2>http://www.cs.indiana.edu/~jjhursey</font></tt></a><tt><font size=2><br>
> <br>
> <br>
> _______________________________________________<br>
> mpi3-ft mailing list<br>
> mpi3-ft@lists.mpi-forum.org<br>
> </font></tt><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft"><tt><font size=2>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</font></tt></a><tt><font size=2><br>
> <br>
> <br>
> <ATT00001..txt><br>
<br>
------------------------------------<br>
Joshua Hursey<br>
Postdoctoral Research Associate<br>
Oak Ridge National Laboratory<br>
</font></tt><a href=http://www.cs.indiana.edu/~jjhursey><tt><font size=2>http://www.cs.indiana.edu/~jjhursey</font></tt></a><tt><font size=2><br>
<br>
<br>
_______________________________________________<br>
mpi3-ft mailing list<br>
mpi3-ft@lists.mpi-forum.org<br>
</font></tt><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft"><tt><font size=2>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</font></tt></a><tt><font size=2><br>
</font></tt>
<br>
<br>