<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">PSB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> mpi3-ft-bounces@lists.mpi-forum.org [mailto:mpi3-ft-bounces@lists.mpi-forum.org]
<b>On Behalf Of </b>Wesley Bland<br>
<b>Sent:</b> Wednesday, July 31, 2013 1:31 PM<br>
<b>To:</b> MPI 3.0 Fault Tolerance and Dynamic Process Control working Group<br>
<b>Subject:</b> Re: [Mpi3-ft] Problem with reusing rendezvous memory buffers<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
</div>
<p><span style="color:#A0A0A8">On Wednesday, July 31, 2013 at 11:50 AM, George Bosilca wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:0in;margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal">On Tue, Jul 30, 2013 at 10:47 PM, Pavan Balaji <<a href="mailto:balaji@mcs.anl.gov">balaji@mcs.anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hmm. Yes, you are right. Generating different per-process rkeys is an<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">option on IB. Though that's obviously less scalable than a single rkey and<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">hurts performance too because a new rkey has to be generated for each<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">process. Even more of a reason for FT to have a requested/provided option<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">like threads.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I fail to understand the scenario allowing you to reach such a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">conclusion. You want to have an MPI_RECV with a buffer where multiple<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">senders can do RMA operations. The only way this can be done in the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">context of the MPI standard is if each of the receives on this<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">particular buffer are using non-contiguous datatypes. Thus, unlike<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">what you suggest in your answer above, this is not hurting performance<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">as you are already in a niche mode (I'm not even talking about the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">fact that usually non-contiguous datatypes conflicts with RMA<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">operations). Moreover, you suppose that the detection of a dead<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">process and the re-posting of the receive buffer can happen faster<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">than an RMA message cross the network. The only potential case where<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">such a scenario can happen is when multiple paths between the source<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">and the destination exist, and the failure detection happen on one<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">path while the RMA message took another one. This is highly improbable<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">in most cases.<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">This isn't necessarily about RMA. This can happen with send/receive as well. I don't disagree that this scenario is unlikely, but it could result in a case where the user ends up with bad data and can't do anything about it. <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">[rich] I think that what you are worried about protecting is an OS that does not provide adequate process protection. I don’t know if we want to go there.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 8.0pt;margin-left:0in;margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">There are too many ifs in this scenario to make it plausible. Even if<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">we suppose that all those ifs will be true, as Rich said, this is an<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">issue of [quality of] implementation not MPI standard. A high quality<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">MPI implementation will delay reporting the process failure error on<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">that particular MPI_RECV until all possible RMA from the dead process<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">were either discarded by the network, or written to the memory.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">However, please also think about this problem for other networks that might<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">not have such hardware protection capabilities (K Computer comes to mind).<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">K computer ? My understanding is that there are such capabilities in<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">the TOFU network. I might be wrong thou, in which case I would<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">definitively appreciate if you can you pinpoint me to a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">link/documentation that proves your point?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Maybe they cannot provide MPI-specified FT, and that would be fine.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Not really, FT can be supported without overhead for the normal<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">execution even for the types of netwrok you mention. The solution I<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">presented above, uses the timeouts of the network layer to ensure no<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">delivery can occur after the error reporting, by delaying the error<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">reporting until all timeout occurred. Trivial to implement, and<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">without impact on the normal execution path.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">George.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-- Pavan<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On 07/30/2013 02:59 PM, Sur, Sayantan wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hi Wesley,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Looks like your attachment didn’t make it through. Using IB, one can<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">generate rkeys for each sender and just invalidate the key for the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">observed failed process. HW can just drop the “slow” message when it<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">arrives. I’m assuming that generating keys should be fast in the future<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">given that recently announced HW/firmware has support for on-demand<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">registration. In any case, it is not a restriction of IB per se.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Sayantan<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">*From:*<a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org">mpi3-ft-bounces@lists.mpi-forum.org</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[<a href="mailto:mpi3-ft-bounces@lists.mpi-forum.org">mailto:mpi3-ft-bounces@lists.mpi-forum.org</a>] *On Behalf Of *Wesley Bland<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">*Sent:* Tuesday, July 30, 2013 11:04 AM<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">*To:* MPI3-FT Working Group<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">*Subject:* [Mpi3-ft] Problem with reusing rendezvous memory buffers<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Pavan pointed out a problem to me yesterday related to memory buffers<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">used with rendezvous protocols. If a process passes a piece of memory to<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">the library in an MPI_RECV and the library gives that memory to the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">hardware, where it is pinned, we can get into trouble if one of the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">processes that could write into that memory fails. The problem comes<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">from a process sending a slow message and then dying. It is possible<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">that the other processes could detect and handle the failure before the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">slow message arrives. Then when the message does arrive, it could<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">corrupt the memory without the application having a way to handle this.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">My whiteboard example is attached as an image.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We can't just unmap memory from the NIC when a failure occurs because<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">that memory is still being used by another process's message. Some<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">hardware supports unmapping memory for specific senders which would<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">solve this issue, but some don't, such as InfiniBand, where the memory<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">region just has a key and unmapping it removes it for all senders.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This problem doesn't have a good solution (that I've come up with), but<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I did come up with a solution. We would need to introduce another error<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">code (something like MPI_ERR_BUFFER_UNUSABLE) that would be able to tell<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">the application that the buffer that the library was using is no longer<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">usable because it might be corrupted. For some hardware, this wouldn't<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">have to be returned, but for hardware where this isn't possible, the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">library could pass this error to the implementation to say that I need a<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">new buffer in order to complete this operation. On the sender side, the<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">operation would probably complete successfully since to it, the memory<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">was still available. That means that there will be some rollback<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">necessary, but that's up to the application to figure out.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I know this is an expensive and painful solution, but this is all I've<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">come up with so far. Thoughts from the group?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Wesley<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">_______________________________________________<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">mpi3-ft mailing list<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:mpi3-ft@lists.mpi-forum.org">mpi3-ft@lists.mpi-forum.org</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</a><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">--<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Pavan Balaji<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://www.mcs.anl.gov/~balaji">http://www.mcs.anl.gov/~balaji</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">_______________________________________________<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">mpi3-ft mailing list<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:mpi3-ft@lists.mpi-forum.org">mpi3-ft@lists.mpi-forum.org</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</a><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">_______________________________________________<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">mpi3-ft mailing list<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="mailto:mpi3-ft@lists.mpi-forum.org">mpi3-ft@lists.mpi-forum.org</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft</a><o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>