<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1993631967;
mso-list-type:hybrid;
mso-list-template-ids:920832792 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">FT WG,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I would like to thank Josh for enduring the marathon plenary presentation! It was truly commendable.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Based on the Forum feedback and vote, it is apparent that there are some significant issues. Primarily due to several new concepts and terms, that the larger Forum does not believe to be required, OR present implementation challenges for
the rest of MPI library.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I would like to argue for a simplified version of the proposal that covers a large percentage of use-cases and resists adding new “features” for the full-range of ABFT techniques. It is good if we have a more pragmatic view and not sacrifice
the entire FT proposal for the 1% fringe cases. Most apps just want to do something like this:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">for(… really long time …) {<o:p></o:p></p>
<p class="MsoNormal"> MPI_Comm_check(work_comm, &is_ok, &alive_group);<o:p></o:p></p>
<p class="MsoNormal"> if(!is_ok) {<o:p></o:p></p>
<p class="MsoNormal"> MPI_Comm_create_group(alive_group, …, &new_comm);<o:p></o:p></p>
<p class="MsoNormal"> // re-balance workload and use new_comm in rest of computation<o:p></o:p></p>
<p class="MsoNormal"> MPI_Comm_free(work_comm); // get rid of old comm<o:p></o:p></p>
<p class="MsoNormal"> work_comm = new_comm;<o:p></o:p></p>
<p class="MsoNormal"> } else {<o:p></o:p></p>
<p class="MsoNormal"> // continue computation using work_comm<o:p></o:p></p>
<p class="MsoNormal"> // if some proc failed in this iteration, roll back work done in this iteration, go back to loop<o:p></o:p></p>
<p class="MsoNormal"> }<o:p></o:p></p>
<p class="MsoNormal">}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here are some modifications I would like to propose to the current chapter (in order as these concepts/terms appear in the text):<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Remove concept of “recognized failed” processes. As was pointed out in the meeting, we don’t really care about the failed processes, rather the alive ones. Accordingly, rename MPI_Comm(win/file)_validate() to MPI_Comm(win/file)_check(MPI_Comm
comm, int * is_ok, MPI_Group * alive_group);<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Remove concept of “collectively inactive/active”. This doesn’t really bring anything to the table, rather conflicts with existing definition of collectives. MPI defines collectives as being equivalent of a series of point-to-point calls.
As per that definition, if the point-to-point calls succeed (i.e. the corresponding processes are alive), then as locally observed, collective call has also succeeded. As far as the application is concerned as long as the local part of collective is complete
successfully, it is OK. If they want to figure out global status, they can always call MPI_Comm_check() or friends.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Eventually perfect failure detector/strongly complete/strongly accurate/etc: We replace this discussion (even remove much of 17.3) with a much more straight-forward requirement – “Communication with a process completes with either success
or error. In case of communication with failed processes, communication calls and requests may complete with MPI_ERR_PROC_FAILSTOP.” Note that MPI standard requires all communication to complete before calling MPI_Finalize – therefore, the first part of this
requirement is nothing new. The second part indicates that there is no guarantee that communication with a failed process *<b>will</b>* fail. Messages may have been internally buffered before the real failure may still be delivered per existing MPI semantics.<o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1">
<![if !supportLists]><span style="mso-list:Ignore">a.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>This does raise the question from implementers: “When do I mark requests as MPI_ERR_PROC_FAILSTOP? How long do I wait?” The answer completely depends on the implementation. Obviously, there is some requirement to deal with process launcher
runtime. In some implementations with connected mode may be able to leverage hw or os techniques to detect connections that have gone down. MPI implementations using connection-less transports may need additional work. However, *<b>none</b>* of this is new
work/concepts. As far as possible, we should talk minimally about what the MPI implementation might do to achieve this.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">4.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Remove process failure handlers – 17.5.1, 17.5.2, 17.5.3, 17.5.4. The only way to find out if something failed is to call MPI_Comm_check() and friends. This removes a whole lot of complexity with failure handlers. Fail handlers can be
emulated over this interface as a library. We may consider them for MPI-3.1 (or 4).<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">5.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Point-to-point communication: Remove the concept of MPI_ERR_ANY_SOURCE_DISABLED and corresponding calls to re-enable any source. The concept of disabling ANY_SOURCE is counter-intuitive. When an app/lib posts a recv with ANY_SOURCE,
it is specifically telling the MPI library that *<b>any</b>* source is OK and implicitly means that if some senders are unable to send, application/lib does not care! Master/slave type of applications wishing to use FT features can periodically call MPI_Comm_check().
Additionally, if the master tries to send to the dead process, it may get an error. My guess is that master/slave type of apps are among the most resilient, and some even work with the current standard (MPI_ERRORS_RETURN). A benefit of removing this restriction
is that we no longer have the threading complexities of re-enabling any source using reader/writer locks
<span style="font-family:Wingdings">J</span> Therefore, we can remove 17.6.3.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">6.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Retain MPI_Comm_drain() and MPI_Comm_idrain() as they provide useful functionality.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">7.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Collective communication: Rename comm_validate() to comm_check() as per discussion above. We can keep comm_check_multiple() as it provides useful functionality for overlapping communicators by reducing overhead to check them. We can
retain much of 17.7.2 while removing references to “collectively inactive”. If the output of collective depends on contribution from a failed process, then obviously, the collective fails. This is in keeping with point-to-point semantics – one cannot receive
any data from a failed process. Keep in mind if the contribution from failed process may have arrived before it failed – and that is OK (not flagged as failure). Some collectives, such as MPI_Bcast, may succeed even if processes down the bcast tree have failed
as sends may simply be buffered. The app/lib will only know if a collective was a global success by either performing an Allreduce after the collective OR calling comm_check(). In any case, it is left to app/lib and not MPI to report failures of processes
the library didn’t try to communicate with during this op.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">8.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>I am proposing that once a collective fails with MPI_ERR_PROC_FAIL_STOP, all subsequent collectives on that comm fail immediately with MPI_ERR_PROC_FAIL_STOP. App/lib needs to use MPI_Comm_create_group() to fork off a new comm of live
procs and continue with it. This is a deviation from the current proposal that allows collectives on bad comms (after re-enabling collectives) and keeps 0s as contributions. I am aware that this might not fully satisfy all use cases (although at this point
of time, I cannot think of any), but in a broader view, we could think this of as a compromise to reduce complexity.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">9.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Example 17.4 changes only slightly to call comm_check() and then split off the new communicator. Why keep failed procs in the communicator anyways?<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">10.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Note that this change in semantics allows us to bypass the question raised: “Why does comm_size() on a communicator with failed procs still return the old value- alive_ranks + failed_ranks?” As I mentioned before, this is odd, and we
should encourage app/lib to only deal with known alive ranks. The current proposal does the reverse – forces app to keep track of “known failed”. This causes confusion!<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">11.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Process topologies 17.9 – should change to say that we can only use communicators with live ranks. i.e. if you know your comm was bad, split off a new comm with live ranks. During the op, some ranks may fail – and that is OK since MPI_ERR_PROC_FAIL_STOP
will be raised. This is mentioned in the current proposal.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">12.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Similar changes in semantics to windows and files.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Please let me know if I overlooked some corner cases or I have mis-interpreted the text of the current chapter. I gave it some thought, but WG knows best!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">===<o:p></o:p></p>
<p class="MsoNormal">Sayantan Sur, Ph.D.<o:p></o:p></p>
<p class="MsoNormal">Intel Corp.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>