<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in'>><br>> Considering many apps work in master-slave mode, I would like to be able to guarantee that the side on which the master resides is not killed.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in'>You should be able to do this by creating subsets of communicators and setting error handers appropriately on the application end. Organize 'workers' into volatile groups (error handler = MPI_ERRORS_ARE_FATAL), while the 'manager' process(es) only ever participate with communicators that do not have a fatal error handler.<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>I don't think I understand. The master and slave need to have a common communicator which will be used to communicate between the two. If the associated error handler is MPI_ERRORS_RETURN and the communication between the two is cut, the MPI lib might decide to kill either of the two ?<o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>If the workers use communicators that are MPI_ERRORS_FATAL, if there is a disconnect with the master, they will be automatically aborted. Meanwhile, the master will be informed about their “failure” because of the disconnect and when connection to the physical nodes that previously hosted the aborted workers is re-established, the master’s MPI library will see that worker tasks are dead and will not need to kill the master.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>Greg Bronevetsky<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>Lawrence Livermore National Lab<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>(925) 424-5756<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>bronevetsky@llnl.gov<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>http://greg.bronevetsky.com <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> mpi3-ft-bounces@lists.mpi-forum.org [mailto:mpi3-ft-bounces@lists.mpi-forum.org] <b>On Behalf Of </b>Toon Knapen<br><b>Sent:</b> Wednesday, February 09, 2011 7:05 AM<br><b>To:</b> MPI 3.0 Fault Tolerance and Dynamic Process Control working Group<br><b>Subject:</b> Re: [Mpi3-ft] run through stabilization user-guide<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><p class=MsoNormal> <o:p></o:p></p></div><p class=MsoNormal>It is important to clarify what we can and cannot specify in the standard. So I appreciate your help with explaining how you think the language can be more precise or better explained.<br><br>For this proposal, we are focused on graceful degradation of the job. Processes will fail(-stop), but the job as a whole is allowed the opportunity to continue operating.<br><br>We are defining process failures as fail-stop failures in the standard. So we cover processes that crash, and do not continue to participate in the parallel environment. How the MPI implementation detects process failure is not defined by the standard, except by the properties that the MPI implementation must provide to the application. The MPI implementation will provide a view of the failure detector that is 'perfect' from the perspective of the application (though internally there is a fair amount of flexibility on how to provide this guarantee).<o:p></o:p></p></blockquote><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal>I understand that processes that are detected (by the MPI lib) to fail will be stopped. That makes sense. What I do worry about is that a process or communication with a specific process is going awry.<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><p class=MsoNormal style='margin-bottom:12.0pt'>It is the responsibility of the application to design around this type of situation to ensure continued progress of their application - making a fault aware application from an otherwise fault unaware application. There are a few ways to do this, but the best solution will always be domain specific.<o:p></o:p></p></blockquote><div><p class=MsoNormal>OK<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><p class=MsoNormal><br>It is possible that the application detects that a peer process is faulty in a different way than fail-stop (e.g., byzantine). For example, a peer process may have incurred a soft-error memory corruption, and is sending invalid data (but valid from the MPI perspective). A peer process could be checking the values, and determine that the peer is faulty. At which point it can either:<br> - Coordinate with the other alive peers to exclude the faulty process, or<br> - Use MPI_Comm_kill() to request that the process be terminated.<br>MPI_Comm_kill() is not described in the user's guide, but is in the main proposal. It allows one process to kill another without killing itself (which would happen if they used MPI_Abort). Is this a scenario that you were concerned about?<o:p></o:p></p></blockquote><div><p class=MsoNormal>Yes, this is important to me. If not having a precise definition of what a 'failed' process actually is (and I understand why you avoid defining it), I'll design the app. such that it might itself decide that some (communication to a) process is failing and thus I want to be able to take the decision myself too. So this makes me feel more comfortable about it.<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><p class=MsoNormal style='margin-bottom:12.0pt'>><br>> Considering many apps work in master-slave mode, I would like to be able to guarantee that the side on which the master resides is not killed.<o:p></o:p></p></div><p class=MsoNormal style='margin-bottom:12.0pt'>You should be able to do this by creating subsets of communicators and setting error handers appropriately on the application end. Organize 'workers' into volatile groups (error handler = MPI_ERRORS_ARE_FATAL), while the 'manager' process(es) only ever participate with communicators that do not have a fatal error handler.<o:p></o:p></p></blockquote><div><p class=MsoNormal>I don't think I understand. The master and slave need to have a common communicator which will be used to communicate between the two. If the associated error handler is MPI_ERRORS_RETURN and the communication between the two is cut, the MPI lib might decide to kill either of the two ?<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><p class=MsoNormal style='margin-bottom:12.0pt'>Does that help clarify?<o:p></o:p></p></blockquote><div><p class=MsoNormal>A lot. Thanks! <o:p></o:p></p></div><div><p class=MsoNormal>I think the way you (and others) clarified this, it would serve the user-guide too to discuss more in-depth what exactly a failure is, why it is not defined in detail and the ability for the app to detect errors and kill processes.<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal>toon<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div></div></div></div></body></html>