<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I think this problem doesn’t exist (even without the per-process key).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The target process will destroy the connection to the “observed failed” process. That itself is sufficient for all networks that I’m aware of to be able to
discard the late message when it arrives. Basically, the late message arrives on a dead connection/endpoint.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Sayantan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> mpi3-ft-bounces@lists.mpi-forum.org [mailto:mpi3-ft-bounces@lists.mpi-forum.org]
<b>On Behalf Of </b>Wesley Bland<br>
<b>Sent:</b> Tuesday, July 30, 2013 1:11 PM<br>
<b>To:</b> MPI 3.0 Fault Tolerance and Dynamic Process Control working Group<br>
<b>Subject:</b> Re: [Mpi3-ft] Problem with reusing rendezvous memory buffers<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">I was afraid the attachment wouldn't go through. I think my point was understandable without it though. No big deal.<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">I didn't know about the ability to generate different keys for each process. That is one way of solving the problem for IB. Is there any other hardware where this might
come up?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Wesley<o:p></o:p></span></p>
</div>
</div>
<p><span style="color:#A0A0A8">On Tuesday, July 30, 2013 at 2:59 PM, Sur, Sayantan wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:0in;margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Wesley,</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Looks like your attachment didn’t make it through. Using IB, one can generate rkeys for each sender and just invalidate the key for the
observed failed process. HW can just drop the “slow” message when it arrives. I’m assuming that generating keys should be fast in the future given that recently announced HW/firmware has support for on-demand registration. In any case, it is not a restriction
of IB per se.</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Sayantan</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p style="margin:0in;margin-bottom:.0001pt"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org">mpi3-ft-bounces@lists.mpi-forum.org</a> [<a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org">mailto:mpi3-ft-bounces@lists.mpi-forum.org</a>]
<b>On Behalf Of </b>Wesley Bland<br>
<b>Sent:</b> Tuesday, July 30, 2013 11:04 AM<br>
<b>To:</b> MPI3-FT Working Group<br>
<b>Subject:</b> [Mpi3-ft] Problem with reusing rendezvous memory buffers</span><o:p></o:p></p>
</div>
</div>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
<div>
<p style="margin:0in;margin-bottom:.0001pt">Pavan pointed out a problem to me yesterday related to memory buffers used with rendezvous protocols. If a process passes a piece of memory to the library in an MPI_RECV and the library gives that memory to the hardware,
where it is pinned, we can get into trouble if one of the processes that could write into that memory fails. The problem comes from a process sending a slow message and then dying. It is possible that the other processes could detect and handle the failure
before the slow message arrives. Then when the message does arrive, it could corrupt the memory without the application having a way to handle this. My whiteboard example is attached as an image.<o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt"> <img border="0" id="_x0000_i1025" src="cid:image001.jpg@01CE8D28.874A12F0"> <o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt">We can't just unmap memory from the NIC when a failure occurs because that memory is still being used by another process's message. Some hardware supports unmapping memory for specific senders which would solve this
issue, but some don't, such as InfiniBand, where the memory region just has a key and unmapping it removes it for all senders.<o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt">This problem doesn't have a good solution (that I've come up with), but I did come up with a solution. We would need to introduce another error code (something like MPI_ERR_BUFFER_UNUSABLE) that would be able to tell
the application that the buffer that the library was using is no longer usable because it might be corrupted. For some hardware, this wouldn't have to be returned, but for hardware where this isn't possible, the library could pass this error to the implementation
to say that I need a new buffer in order to complete this operation. On the sender side, the operation would probably complete successfully since to it, the memory was still available. That means that there will be some rollback necessary, but that's up to
the application to figure out.<o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt">I know this is an expensive and painful solution, but this is all I've come up with so far. Thoughts from the group?<o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt"> <o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt">Thanks,<o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt">Wesley<o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal">_______________________________________________<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">mpi3-ft mailing list<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:mpi3-ft@lists.mpi-forum.org">mpi3-ft@lists.mpi-forum.org</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</a><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>