<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Toon, thank you for the feedback! Let me try to answer your questions, though it would also be great to hear from Josh who’s now the guiding spirit of the group.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>The spec explicitly doesn’t define the meaning of failure because it is a very low-level concept. The MPI implementation is responsible for detecting failures and defining what qualifies as one. The only guarantee that you can rely on is that if the “failure” is bad enough that a given MPI rank can’t communicate with others, then MPI will have to either abort the application completely or eventually report this failure via the FT API. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Note that this view is focused on process failures. When it is applied to things like network partitions (this includes the case you mentioned where one process can’t talk to any other due to a failed network card) then processes on both parts of the partition may be informed that the others have failed. As such, when connectivity is restored, since MPI will be responsible to maintaining self-consistency of its previous notifications, it’ll have to kill processes on one side of the partition to keep consistent with the notifications it gave to the other partition. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>We haven’t talked about the process itself detecting failures independently of the MPI library. Right now the only thing a process can do is call MPI_Abort() if it notices that its core is not operating correctly. However, in the future I can easily see us providing applications with call via which they can inform MPI about network failures and failures of other processes.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>Greg Bronevetsky<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>Lawrence Livermore National Lab<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>(925) 424-5756<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>bronevetsky@llnl.gov<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>http://greg.bronevetsky.com <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> mpi3-ft-bounces@lists.mpi-forum.org [mailto:mpi3-ft-bounces@lists.mpi-forum.org] <b>On Behalf Of </b>Toon Knapen<br><b>Sent:</b> Sunday, February 06, 2011 10:19 AM<br><b>To:</b> mpi3-ft@lists.mpi-forum.org<br><b>Subject:</b> [Mpi3-ft] run through stabilization user-guide<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Hi all,<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal>(I've ended up on this ml because of a recent discussion on the OpenMPI ml referring to this user-guide and because I'm working on a parallel financial app that runs in batch-mode but will also be run in online-mode. The calculations are only loosely coupled calculations but stability/fault-tolerance is important when running in batch-mode and certainly when running online)<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal>So I ended up reading the user guide (<a href="https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_users_guide" target="_blank">https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_users_guide</a>) and since it seems destined to app developers like me, I'll provide some feedback from my pov:<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal>In the motivation section, paragraph 4: Hopefully not a silly question but what is actually considered to be a failure ? Looking into the archives of this ml there are many discussions on the topic but what is actually the current conclusion?<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Additionally, is it possible that a process can detect that the process itself fails, e.g. if its network went down and thus unable to communicate any further.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Finally, I think there is a typo in the first example in the section Point-to-Point operators. It relies on 'status' for knowing the failed rank while IMO it should be 'peer'.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>toon<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div></div></div></body></html>