Tony,<div><br></div><div>I have to admit that I am getting a bit lost in your multipart presentation of your proposal. So that is making it difficult for me to reason about it as a whole. Can you synthesis your proposal (either as a separate email thread or on the wiki) and maybe we can discuss it further on the teleconf and mailing list? That will give us a single place to point to and refine during the subsequent discussions.</div>
<div><br></div><div>Thanks,</div><div>Josh<br><br><div class="gmail_quote">On Mon, Jan 16, 2012 at 4:18 PM, Anthony Skjellum <span dir="ltr"><<a href="mailto:tony@runtimecomputing.com">tony@runtimecomputing.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>More</div><div><br></div><div>#5)</div><div><br></div><div>A parallel thread, running on the same group as the communicator used for the user program in FTBLOCK, would be free to be running a gossip protocol,</div>
<div>
instantiated</div><div>a) by the user at his/her choice</div><div>b) by MPI at its choice, by user demand</div><div><br></div><div>This parallel inspector thread propagates error state, and accepts input to the error state by allowing user programs to assert error state within the FTBLOCK.</div>
<div><br></div><div>You can think of this parallel thread as cooperation of the progress engines with gossip, or as a completely optional activity, created by the user, or as </div><div>a part of the FT proposal. As long as there are local ways to add error state, we can write this as a layered library for the sake of making sure that local</div>
<div>errors are propagated (remembering that MPI is returning local errors to the user inside his/her FTBLOCK).</div><div><br></div><div>If we want the implementation to provide/support this kind of FT-friendly error propagation, that is a bigger step, which I am not advocating as required.</div>
<div>I think it would be nice to have this be part of a fault tolerant communication space. But, I am OK with users doing this for themselves, esp. because they</div><div>can control granularity better.</div><div><br></div>
<div>#6) I think MPI_COMM_WORLD is a bummer in the FT world. We don't want it to hang around very long after we get going. If we are really working on</div><div>subgroups, and these subgroups form a hierarchical graph, rather than an all-to-all virtual topology, I don't want to reconstruct MPI_COMM_WORLD after</div>
<div>the first process breaks. So - as an example of pain of MPI scalability - the build-down model of MPI-1 is less good for this than a build-up</div><div>model, where we find a way for groups to rendezvous. Obviously, for convenience, the all-to-all virtual topology in the onset of MPI_Init() is nice,</div>
<div>but I am assuming that errors may happen quite quickly. </div><div><br></div><div>MPI_Init() with no failures is only good for a limited window of time, given failure rates. During this time, we would like to "turn off MPI_COMM_WORLD"</div>
<div>unless we really don't need it, or at least not have to reconstruct it ever if we don't need it.</div><div><br></div><div>So, we need to agree on a fault-friendly MPI_Init()...</div><div><br></div><div>An FTBLOCK-like idea surrounding MPI_Init(), where the external rules of spawning MPI, or the spawn command that creates the world, create an effective communicator. We should generalize MPI_Init() to support outcomes other than pure success, such as partial success (e.g., smaller world than stipulated).</div>
<div><br></div><div>A really good approach would be to only create the actual communicators you really need to run your process, not the virtual-all-to-all world of MPI-1 and MPI-2.</div><div>But, that is a heresy, so forgive me.</div>
<div><br></div><div>Tony</div><div class="HOEnZb"><div class="h5"><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br><br><div class="gmail_quote">On Mon, Jan 16, 2012 at 2:46 PM, Anthony Skjellum <span dir="ltr"><<a href="mailto:tony@runtimecomputing.com" target="_blank">tony@runtimecomputing.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Satantan,<div><br></div><div>Here are my further thoughts, which hopefully includes an answer to your questions :-)</div>
<div><br></div><div>1) Everything is set to "ERRORS return"; the goal is to get local errors back if they are available. </div>
<div><br></div><div>2) I would emphasize non-blocking operations, but blocking operations implemented with internal timeout could return a timeout-type error</div><div><br></div><div>3) You don't have to return the same return code, or the same results in all processes in the communicator, you can get erroneous results</div>
<div> or local failures; the functions are also allowed to produce incorrect results [and we should then discuss what error reporting means here...</div><div> I am happy with local errors returned where known, recognizing those processes may die before the bottom of the block. However, I also</div>
<div> expect the implementation to do its best to propagate error state knowledge within this FTBLOCK, based organically on ongoing communication</div><div> or on gossip if an implementation so chose.]</div><div><br>
</div><div> Also, because we assume that there is also algorithmic fault tolerance at work, local errors may be raised by the application because it is doing checks</div><div> for validity etc.</div><div><br></div><div>
So, either MPI or the application may raise local errors prior to the bottom of the FTBLOCK, and the success of the bottom of the block must be allowed</div><div> to be fail, just based on ABFT inputs from the application to MPI, not just based on MPI's opinion.</div>
<div><br></div><div>4) If you are willing to do everything nonblocking, then I can describe the test at the bottom of the FTBLOCK as follows:</div><div><br></div><div>The test operation at the bottom of the FTBLOCK is effectively a generalized WAIT_ALL, that completes or doesn't complete all the outstanding</div>
<div>requests, returning errors related to the faults observed, and providing a unified 0/1 success failure state consistently across the group of comm</div><div>[or surviving members thereof]. </div><div><br></div><div>
In my view, the application as well as MPI can contribute error state as input to the FTBLOCK test.</div><div><br></div><div>Also, the application that gets local errors inside the loop already is immediately ready to punt at that point, and do a jump to the bottom of the loop. Let's </div>
<div>assume it is required to do all the operations before getting to the bottom of the loop, for now, and we just allow that some of these may return further</div><div>errors (I am trying to keep it simple), and MPI-like rules of all attempting the operations. If we can get this to work, we can weaken it later.</div>
<div><br></div><div>There is no good way to describe some BLOCKING, some nonblocking operations, because we have no descriptor to tell us if something failed that previously</div><div>returned... and did not give a local error, so I am not going to pursue BLOCKING for now. Let's assume we cannot do BLOCKING, and weaken this later,</div>
<div>if we can get a consistent solution using all nonblocking. </div><div><br></div><div>Please tell me what you think.</div><div><br></div><div>Thanks for responding!</div><div><br></div><div>Tony</div><div>
<div><div><br></div><div>
<br><div class="gmail_quote">On Mon, Jan 16, 2012 at 12:39 PM, Sur, Sayantan <span dir="ltr"><<a href="mailto:sayantan.sur@intel.com" target="_blank">sayantan.sur@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Tony,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">In the example semantics you mentioned, are the “ops” required to return the same result on all processors? Although this doesn’t change the API “op”, but it
does change completion semantics of almost all MPI ops. I hope I am correctly interpreting your message.<u></u><u></u></span></p><div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">===</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Sayantan Sur, Ph.D.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Intel Corp.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
</div><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org" target="_blank">mpi3-ft-bounces@lists.mpi-forum.org</a> [mailto:<a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org" target="_blank">mpi3-ft-bounces@lists.mpi-forum.org</a>]
<b>On Behalf Of </b>Anthony Skjellum<br>
<b>Sent:</b> Sunday, January 15, 2012 7:06 PM</span></p><div><div><br>
<b>To:</b> MPI 3.0 Fault Tolerance and Dynamic Process Control working Group<br>
<b>Subject:</b> Re: [Mpi3-ft] simplified FT proposal<u></u><u></u></div></div><p></p>
</div>
</div><div><div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Everyone, I think we need to start from scratch.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We should look for minimal fault-tolerant models that are achievable and useful. They may allow for a combination of faults (process and network), but in the end, as discussed in San Jose:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">FTBLOCK<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">--------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Start_Block(comm)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">op [normal MPI operation on communicator specified by Start_Block, or subset thereof]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">op<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">op<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Test_Block(comm)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Which either succeeds, or fails, on the whole list of operations, followed by ways to reconstruct communicators and add back processes (easily),<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">provides for 3-level fault tolerant model<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">a) Simply retry if the kind of error at the Test_Block is retryable<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">b) Simply reconstruct the communicator, use algorithmic fault tolerance to get lost data, and retry the block<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">c) drop back to 1 or more levels of checkpoint-restart.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We can envision in this model an unrolling of work, in terms of a parameter N, if there is a lot of vector work, to allow granularity control<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">as a function of fault environment.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">In some sense, a simpler model, that provides for efforts to detect<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">a) by MPI<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">b) allowing application monitor to assert failure asynchronously to this loop<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">provides a more general ability to have coverage of faults, including but not limited to process faults and possible network faults.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">It changes the API very little.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">It motivates the use of buffers, not zero copy, to support the fact that you may to roll-back a series of operations, thereby revealing fault-free overhead directly.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Start_Block and Test_Block are collective and synchronizing, such as barriers. Because we have uncertainty to within a message, multiple barriers (as mentioned by George Bosilca K to me in a sidebar at the meeting).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We try to get this to work, COMPLETELY, and ratify this in MPI 3.x, if we can. Once we have this stable intermediate form, we explore more<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">options.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I think it is important to recognize that the reconstruction step, including readding processes and making new communicators may mean smarter Join operations. It is clear we need to be able to treat failures during the recovery process,
and use a second level loop, possibly bombing out<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">to checkpoint, if we cannot make net progress on recovery because of unmodeled error issues.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The testing part leverages all the learning so far, but needn't be restricted to modeled errors like process faults. There can be modeled and unmodeled faults. Based on what fault comes up, the user application then has to decide how
hard a retry to do, whether just to add processes,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">whether just to retry the loop, whether to go to a checkpoint, whether to restart the APP. MPI could give advice, based on its understanding of<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">the fault model, in terms of sufficient conditions for "working harder" vs "trying the easiest" for fault models it understands somewhat for a system.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Now, the comments here are a synopsis of part of the side bars and open discussion we had in San Jose, distilled a bit. I want to know why <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">we can't start with this, succeed with this, implement and test it, and having succeeded, do more in a future 3.y, y > x release, given user experience.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I am not speaking to the choice of "killing all communicators" as with FT-MPI, or "just remaking those you need to remake." I think we need to resolve. Honestly, groups own the fault property, not communicators, and all groups held by
communicators where the fault happened should be rebuilt, not all communicators... Let's argue on that.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So, my suggestion is REBOOT the proposal with something along lines above, unless you all see this is no better.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">With kind regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Tony<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Sun, Jan 15, 2012 at 8:00 PM, Sur, Sayantan <<a href="mailto:sayantan.sur@intel.com" target="_blank">sayantan.sur@intel.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Bill,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I am in agreement with your suggestion to have a collective over a communicator that returns a new
communicator containing ranks “alive some point during construction”. It provides cleaner semantics. The example was merely trying to utilize the new MPI_Comm_create_group API that the Forum is considering.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">MPI_Comm_check provides a method to form global consensus in that all ranks in comm did call it.
It does not imply anything about current status of comm, or even the status “just before” the call returns. During the interval before the next call to MPI_Comm_check, it is possible that many ranks fail. However, the app/lib using MPI knows the point where
everyone was alive.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks.</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">===</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Sayantan Sur, Ph.D.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Intel Corp.</span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
</div>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org" target="_blank">mpi3-ft-bounces@lists.mpi-forum.org</a> [mailto:<a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org" target="_blank">mpi3-ft-bounces@lists.mpi-forum.org</a>]
<b>On Behalf Of </b>William Gropp<br>
<b>Sent:</b> Sunday, January 15, 2012 2:41 PM<br>
<b>To:</b> MPI 3.0 Fault Tolerance and Dynamic Process Control working Group<br>
<b>Subject:</b> Re: [Mpi3-ft] simplified FT proposal</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">One concern that I have with fault tolerant proposals has to do with races in the specification. This is an area where users often "just want it to work" but getting it right is
tricky. In the example here, the "alive_group" is really only that at some moment shortly before "MPI_Comm_check" returns (and possibly not even that). After that, it is really the "group_of_processes_that_was_alive_at_some_point_in_the_past". Since there
are sometimes correlations in failures, this could happen even if the initial failure is rare. An alternate form might be to have a routine, collective over a communicator, that returns a new communicator meeting some definition of "members were alive at
some point during construction". It wouldn't guarantee you could use it, but it would have cleaner semantics.<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Bill<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">I would like to argue for a simplified version of the proposal that covers a large percentage of use-cases and
resists adding new “features” for the full-range of ABFT techniques. It is good if we have a more pragmatic view and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps just want to do something like this:</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">for(… really long time …) {</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> MPI_Comm_check(work_comm, &is_ok, &alive_group);</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> if(!is_ok) {</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> MPI_Comm_create_group(alive_group, …, &new_comm);</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> // re-balance workload and use new_comm in rest of computation</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> MPI_Comm_free(work_comm); // get rid of old comm</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> work_comm = new_comm;</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> } else {</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> // continue computation using work_comm</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> // if some proc failed in this iteration, roll back work done in this iteration, go back to loop</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> }</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">}</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> </span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">William Gropp</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">Director, Parallel Computing Institute</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">Deputy Director for Research</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">Institute for Advanced Computing Applications and Technologies</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">Paul and Cynthia Saylor Professor of Computer Science</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">University of Illinois Urbana-Champaign</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><br>
_______________________________________________<br>
mpi3-ft mailing list<br>
<a href="mailto:mpi3-ft@lists.mpi-forum.org" target="_blank">mpi3-ft@lists.mpi-forum.org</a><br>
<a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">-- <br>
Tony Skjellum, PhD<br>
RunTime Computing Solutions, LLC<br>
<a href="mailto:tony@runtimecomputing.com" target="_blank">tony@runtimecomputing.com</a><br>
direct: <a href="tel:%2B1-205-314-3595" value="+12053143595" target="_blank">+1-205-314-3595</a><br>
cell: <a href="tel:%2B1-205-807-4968" value="+12058074968" target="_blank">+1-205-807-4968</a><u></u><u></u></p>
</div>
</div></div></div>
</div>
</div>
<br>_______________________________________________<br>
mpi3-ft mailing list<br>
<a href="mailto:mpi3-ft@lists.mpi-forum.org" target="_blank">mpi3-ft@lists.mpi-forum.org</a><br>
<a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Tony Skjellum, PhD<br>
RunTime Computing Solutions, LLC<br><a href="mailto:tony@runtimecomputing.com" target="_blank">tony@runtimecomputing.com</a><br>direct: <a href="tel:%2B1-205-314-3595" value="+12053143595" target="_blank">+1-205-314-3595</a><br>
cell: <a href="tel:%2B1-205-807-4968" value="+12058074968" target="_blank">+1-205-807-4968</a><br><br>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Tony Skjellum, PhD<br>RunTime Computing Solutions, LLC<br><a href="mailto:tony@runtimecomputing.com" target="_blank">tony@runtimecomputing.com</a><br>
direct: <a href="tel:%2B1-205-314-3595" value="+12053143595" target="_blank">+1-205-314-3595</a><br>
cell: <a href="tel:%2B1-205-807-4968" value="+12058074968" target="_blank">+1-205-807-4968</a><br><br>
</div>
</div></div><br>_______________________________________________<br>
mpi3-ft mailing list<br>
<a href="mailto:mpi3-ft@lists.mpi-forum.org">mpi3-ft@lists.mpi-forum.org</a><br>
<a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Joshua Hursey<br>
Postdoctoral Research Associate<br>Oak Ridge National Laboratory<br><a href="http://users.nccs.gov/~jjhursey" target="_blank">http://users.nccs.gov/~jjhursey</a><br>
</div>