<div>More</div><div><br></div><div>#5)</div><div><br></div><div>A parallel thread, running on the same group as the communicator used for the user program in FTBLOCK, would be free to be running a gossip protocol,</div><div>
instantiated</div><div>a) by the user at his/her choice</div><div>b) by MPI at its choice, by user demand</div><div><br></div><div>This parallel inspector thread propagates error state, and accepts input to the error state by allowing user programs to assert error state within the FTBLOCK.</div>
<div><br></div><div>You can think of this parallel thread as cooperation of the progress engines with gossip, or as a completely optional activity, created by the user, or as�</div><div>a part of the FT proposal. �As long as there are local ways to add error state, we can write this as a layered library for the sake of making sure that local</div>
<div>errors are propagated (remembering that MPI is returning local errors to the user inside his/her FTBLOCK).</div><div><br></div><div>If we want the implementation to provide/support this kind of FT-friendly error propagation, that is a bigger step, which I am not advocating as required.</div>
<div>I think it would be nice to have this be part of a fault tolerant communication space. �But, I am OK with users doing this for themselves, esp. because they</div><div>can control granularity better.</div><div><br></div>
<div>#6) I think MPI_COMM_WORLD is a bummer in the FT world. �We don't want it to hang around very long after we get going. �If we are really working on</div><div>subgroups, and these subgroups form a hierarchical graph, rather than an all-to-all virtual topology, I don't want to reconstruct MPI_COMM_WORLD after</div>
<div>the first process breaks. �So - as an example of pain of MPI scalability - the build-down model of MPI-1 is less good for this than a build-up</div><div>model, where we find a way for groups to rendezvous. �Obviously, for convenience, the all-to-all virtual topology in the onset of MPI_Init() is nice,</div>
<div>but I am assuming that errors may happen quite quickly. �</div><div><br></div><div>MPI_Init() with no failures is only good for a limited window of time, given failure rates. �During this time, we would like to "turn off MPI_COMM_WORLD"</div>
<div>unless we really don't need it, or at least not have to reconstruct it ever if we don't need it.</div><div><br></div><div>So, we need to agree on a fault-friendly MPI_Init()...</div><div><br></div><div>An FTBLOCK-like idea surrounding MPI_Init(), where the external rules of spawning MPI, or the spawn command that creates the world, create an effective communicator. �We should generalize MPI_Init() to support outcomes other than pure success, such as partial success (e.g., smaller world than stipulated).</div>
<div><br></div><div>A really good approach would be to only create the actual communicators you really need to run your process, not the virtual-all-to-all world of MPI-1 and MPI-2.</div><div>But, that is a heresy, so forgive me.</div>
<div><br></div><div>Tony</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br><br><div class="gmail_quote">On Mon, Jan 16, 2012 at 2:46 PM, Anthony Skjellum <span dir="ltr"><<a href="mailto:tony@runtimecomputing.com">tony@runtimecomputing.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Satantan,<div><br></div><div>Here are my further thoughts, which hopefully includes an answer to your questions :-)</div>
<div><br></div><div>1) Everything is set to "ERRORS return"; the goal is to get local errors back if they are available. �</div>
<div><br></div><div>2) I would emphasize non-blocking operations, but blocking operations implemented with internal timeout could return a timeout-type error</div><div><br></div><div>3) You don't have to return the same return code, or the same results in all processes in the communicator, you can get erroneous results</div>
<div>� � or local failures; the functions are also allowed to produce incorrect results [and we should then discuss what error reporting means here...</div><div>� � I am happy with local errors returned where known, recognizing those processes may die before the bottom of the block. �However, I also</div>
<div>� � expect the implementation to do its best to propagate error state knowledge within this FTBLOCK, based organically on ongoing communication</div><div>� � or on gossip if an implementation so chose.]</div><div><br>
</div><div>� �Also, because we assume that there is also algorithmic fault tolerance at work, local errors may be raised by the application because it is doing checks</div><div>� �for validity etc.</div><div><br></div><div>
� So, either MPI or the application may raise local errors prior to the bottom of the FTBLOCK, and the success of the bottom of the block must be allowed</div><div>� to be fail, just based on ABFT inputs from the application to MPI, not just based on MPI's opinion.</div>
<div><br></div><div>4) If you are willing to do everything nonblocking, then I can describe the test at the bottom of the FTBLOCK as follows:</div><div><br></div><div>The test operation at the bottom of the FTBLOCK is effectively a generalized WAIT_ALL, that completes or doesn't complete all the outstanding</div>
<div>requests, returning errors related to the faults observed, and providing a unified 0/1 success failure state consistently across the group of comm</div><div>[or surviving members thereof]. �</div><div><br></div><div>
In my view, the application as well as MPI can contribute error state as input to the FTBLOCK test.</div><div><br></div><div>Also, the application that gets local errors inside the loop already is immediately ready to punt at that point, and do a jump to the bottom of the loop. �Let's�</div>
<div>assume it is required to do all the operations before getting to the bottom of the loop, for now, and we just allow that some of these may return further</div><div>errors (I am trying to keep it simple), and MPI-like rules of all attempting the operations. �If we can get this to work, we can weaken it later.</div>
<div><br></div><div>There is no good way to describe some BLOCKING, some nonblocking operations, because we have no descriptor to tell us if something failed that previously</div><div>returned... and did not give a local error, so I am not going to pursue BLOCKING for now. �Let's assume we cannot do BLOCKING, and weaken this later,</div>
<div>if we can get a consistent solution using all nonblocking. �</div><div><br></div><div>Please tell me what you think.</div><div><br></div><div>Thanks for responding!</div><div><br></div><div>Tony</div><div class="HOEnZb">
<div class="h5"><div><br></div><div>
<br><div class="gmail_quote">On Mon, Jan 16, 2012 at 12:39 PM, Sur, Sayantan <span dir="ltr"><<a href="mailto:sayantan.sur@intel.com" target="_blank">sayantan.sur@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Tony,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u>�<u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">In the example semantics you mentioned, are the �ops� required to return the same result on all processors? Although this doesn�t change the API �op�, but it
does change completion semantics of almost all MPI ops. I hope I am correctly interpreting your message.<u></u><u></u></span></p><div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u>�<u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u>�<u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">===</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Sayantan Sur, Ph.D.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Intel Corp.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u>�<u></u></span></p>
</div><div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org" target="_blank">mpi3-ft-bounces@lists.mpi-forum.org</a> [mailto:<a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org" target="_blank">mpi3-ft-bounces@lists.mpi-forum.org</a>]
<b>On Behalf Of </b>Anthony Skjellum<br>
<b>Sent:</b> Sunday, January 15, 2012 7:06 PM</span></p><div><div><br>
<b>To:</b> MPI 3.0 Fault Tolerance and Dynamic Process Control working Group<br>
<b>Subject:</b> Re: [Mpi3-ft] simplified FT proposal<u></u><u></u></div></div><p></p>
</div>
</div><div><div>
<p class="MsoNormal"><u></u>�<u></u></p>
<p class="MsoNormal">Everyone, I think we need to start from scratch.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">We should look for minimal fault-tolerant models that are achievable and useful. �They may allow for a combination of faults (process and network), but in the end, as discussed in San Jose:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">FTBLOCK<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">--------------<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Start_Block(comm)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">op [normal MPI operation on communicator specified by Start_Block, or subset thereof]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">op<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">op<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">Test_Block(comm)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">Which either succeeds, or fails, on the whole list of operations, followed by ways to reconstruct communicators and add back processes (easily),<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">provides for 3-level fault tolerant model<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">a) Simply retry if the kind of error at the Test_Block is retryable<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">b) Simply reconstruct the communicator, use algorithmic fault tolerance to get lost data, and retry the block<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">c) drop back to 1 or more levels of checkpoint-restart.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">We can envision in this model an unrolling of work, in terms of a parameter N, if there is a lot of vector work, to allow granularity control<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">as a function of fault environment.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">In some sense, a simpler model, that provides for efforts to detect<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">a) by MPI<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">b) allowing application monitor to assert failure asynchronously to this loop<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">provides a more general ability to have coverage of faults, including but not limited to process faults and possible network faults.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">It changes the API very little.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">It motivates the use of buffers, not zero copy, to support the fact that you may to roll-back a series of operations, thereby revealing fault-free overhead directly.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">Start_Block and Test_Block are collective and synchronizing, such as barriers. �Because we have uncertainty to within a message, multiple barriers (as mentioned by George Bosilca K to me in a sidebar at the meeting).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">We try to get this to work, COMPLETELY, and ratify this in MPI 3.x, if we can. �Once we have this stable intermediate form, we explore more<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">options.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">I think it is important to recognize that the reconstruction step, including readding processes and making new communicators may mean smarter Join operations. �It is clear we need to be able to treat failures during the recovery process,
and use a second level loop, possibly bombing out<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">to checkpoint, if we cannot make net progress on recovery because of unmodeled error issues.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">The testing part leverages all the learning so far, but needn't be restricted to modeled errors like process faults. �There can be modeled and unmodeled faults. �Based on what fault comes up, the user application then has to decide how
hard a retry to do, whether just to add processes,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">whether just to retry the loop, whether to go to a checkpoint, whether to restart the APP. �MPI could give advice, based on its understanding of<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">the fault model, in terms of sufficient conditions for "working harder" vs "trying the easiest" for fault models it understands somewhat for a system.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">Now, the comments here are a synopsis of part of the side bars and open discussion we had in San Jose, distilled a bit. �I want to know why�<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">we can't start with this, succeed with this, implement and test it, and having succeeded, do more in a future 3.y, y > x release, given user experience.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">I am not speaking to the choice of "killing all communicators" as with FT-MPI, or "just remaking those you need to remake." �I think we need to resolve. �Honestly, groups own the fault property, not communicators, and all groups held by
communicators where the fault happened should be rebuilt, not all communicators... �Let's argue on that.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">So, my suggestion is REBOOT the proposal with something along lines above, unless you all see this is no better.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal">With kind regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Tony<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
<div>
<p class="MsoNormal">On Sun, Jan 15, 2012 at 8:00 PM, Sur, Sayantan <<a href="mailto:sayantan.sur@intel.com" target="_blank">sayantan.sur@intel.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Bill,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">�</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I am in agreement with your suggestion to have a collective over a communicator that returns a new
communicator containing ranks �alive some point during construction�. It provides cleaner semantics. The example was merely trying to utilize the new MPI_Comm_create_group API that the Forum is considering.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">�</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">MPI_Comm_check provides a method to form global consensus in that all ranks in comm did call it.
It does not imply anything about current status of comm, or even the status �just before� the call returns. During the interval before the next call to MPI_Comm_check, it is possible that many ranks fail. However, the app/lib using MPI knows the point where
everyone was alive.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">�</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks.</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">�</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">===</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Sayantan Sur, Ph.D.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Intel Corp.</span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">�</span><u></u><u></u></p>
</div>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org" target="_blank">mpi3-ft-bounces@lists.mpi-forum.org</a> [mailto:<a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org" target="_blank">mpi3-ft-bounces@lists.mpi-forum.org</a>]
<b>On Behalf Of </b>William Gropp<br>
<b>Sent:</b> Sunday, January 15, 2012 2:41 PM<br>
<b>To:</b> MPI 3.0 Fault Tolerance and Dynamic Process Control working Group<br>
<b>Subject:</b> Re: [Mpi3-ft] simplified FT proposal</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">�<u></u><u></u></p>
<p class="MsoNormal">One concern that I have with fault tolerant proposals has to do with races in the specification. �This is an area where users often "just want it to work" but getting it right is
tricky. �In the example here, the "alive_group" is really only that at some moment shortly before "MPI_Comm_check" returns (and possibly not even that). �After that, it is really the "group_of_processes_that_was_alive_at_some_point_in_the_past". �Since there
are sometimes correlations in failures, this could happen even if the initial failure is rare. �An alternate form might be to have a routine, collective over a communicator, that returns a new communicator meeting some definition of "members were alive at
some point during construction". �It wouldn't guarantee you could use it, but it would have cleaner semantics.<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">�<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Bill<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">�<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Jan 13, 2012, at 3:41 PM, Sur, Sayantan wrote:<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u>�<u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">I would like to argue for a simplified version of the proposal that covers a large percentage of use-cases and
resists adding new �features� for the full-range of ABFT techniques. It is good if we have a more pragmatic view and not sacrifice the entire FT proposal for the 1% fringe cases. Most apps just want to do something like this:</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">�</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">for(� really long time �) {</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">�� MPI_Comm_check(work_comm, &is_ok, &alive_group);</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">�� if(!is_ok) {</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">� �����MPI_Comm_create_group(alive_group, �, &new_comm);</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">� ����// re-balance workload and use new_comm in rest of computation</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">������ MPI_Comm_free(work_comm); // get rid of old comm</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">������ work_comm = new_comm;</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">�� } else {</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">� ���// continue computation using work_comm</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">���� // if some proc failed in this iteration, roll back work done in this iteration, go back to loop</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">�� }</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">}</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">�</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">�<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">William�Gropp</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">Director, Parallel Computing Institute</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">Deputy Director for Research</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">Institute for Advanced Computing Applications and Technologies</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">Paul and Cynthia Saylor Professor of Computer Science</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">University of Illinois Urbana-Champaign</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica","sans-serif"">�</span><u></u><u></u></p>
</div>
<p class="MsoNormal">�<u></u><u></u></p>
</div>
<p class="MsoNormal">�<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><br>
_______________________________________________<br>
mpi3-ft mailing list<br>
<a href="mailto:mpi3-ft@lists.mpi-forum.org" target="_blank">mpi3-ft@lists.mpi-forum.org</a><br>
<a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u>�<u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">-- <br>
Tony Skjellum, PhD<br>
RunTime Computing Solutions, LLC<br>
<a href="mailto:tony@runtimecomputing.com" target="_blank">tony@runtimecomputing.com</a><br>
direct: <a href="tel:%2B1-205-314-3595" value="+12053143595" target="_blank">+1-205-314-3595</a><br>
cell: <a href="tel:%2B1-205-807-4968" value="+12058074968" target="_blank">+1-205-807-4968</a><u></u><u></u></p>
</div>
</div></div></div>
</div>
</div>
<br>_______________________________________________<br>
mpi3-ft mailing list<br>
<a href="mailto:mpi3-ft@lists.mpi-forum.org" target="_blank">mpi3-ft@lists.mpi-forum.org</a><br>
<a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Tony Skjellum, PhD<br>
RunTime Computing Solutions, LLC<br><a href="mailto:tony@runtimecomputing.com" target="_blank">tony@runtimecomputing.com</a><br>direct: <a href="tel:%2B1-205-314-3595" value="+12053143595" target="_blank">+1-205-314-3595</a><br>
cell: <a href="tel:%2B1-205-807-4968" value="+12058074968" target="_blank">+1-205-807-4968</a><br><br>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Tony Skjellum, PhD<br>RunTime Computing Solutions, LLC<br><a href="mailto:tony@runtimecomputing.com">tony@runtimecomputing.com</a><br>direct: +1-205-314-3595<br>
cell: +1-205-807-4968<br><br>
</div>