<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'>
<p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'>I think the discussion of the three possible
implementations below misses the mark a bit. Any low level messaging
layer (DCMF, LAPI, even "MPI") that can optimize the allreduce and
barrier operations could implement the allfenceall operation as described
below. Even though DCMF - which what MPICH2 on BG/P is based on - uses
the BG/P collective network, it still is *only* a fast allreduce that has
benefits over other pt2pt allreduce algorithms and not very interesting when
considering how an allfenceall would be implemented. </span><br>
<br>
<span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>If we look at
any exascale goal(s), as Jeff mentioned earlier, we'll see that any time an
application or middleware needs to track counters we won't get the scaling
required (assuming the allfenceall is flexible and not restricted to only
"comm_world" operations). What is needed is an interface that allows
low level software to take advantage of any hardware or network features that
would enable a "counter-less" allfenceall implementation. On BG/P,
this would probably take advantage of features such as the torus dimensions of
the network, the (configurable) deterministic packet routing, the "deposit
bit" in the packet header, etc. I'm not saying I have any specific design
in mind, just that experience tells me that we should be able to do something
cool in this space supported by the BG hardware which would eliminate the need
for software accounting.</span> <br>
<br>
<span style='color:#1F497D'><o:p></o:p></span></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:11.0pt;
font-family:"Calibri","sans-serif";color:#E36C0A'>The forum was relatively
clear in requiring an implementation for new calls. In theory, that
implementation kind of has to fit with the motivation for the call.
Saying “this would give performance advantages if we did something special,
but this implementation doesn’t give those advantages” isn’t
particularly compelling. The implementations you mention from
earlier in the thread could colloquially be called “the way it is done
now”, “the way IBM suggested”, and “the way Keith
suggested that may be completely ridiculous ”. Better
implementations would certainly be interesting to discuss.<o:p></o:p></span></b></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:11.0pt;
font-family:"Calibri","sans-serif";color:#E36C0A'>BTW, in many of the RMA
discussions, it has been asserted that using the deterministic packet routing is
a terrible performance hit. Is that not an issue?<o:p></o:p></span></b></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
<span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>If there was an
allfenceall interface, then low level message could do whatever it takes to get
performance and scaling. Without the allfenceall interface middleware is
forced to use one of these platform independent algorithms (allreduce, barrier,
etc) and accept any inherent scaling and performance restrictions.</span> <o:p></o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:11.0pt;
font-family:"Calibri","sans-serif";color:#E36C0A'>I don’t understand this
comment. Without the allfenceall, I’m not sure how you would call a
collective to do a fenceall at a given node (fenceall would complete all
outstanding requests from a given rank). I’m also not
sure why the difference between an allfenceall and a fenceall makes a
difference between platform specific and platform independent approaches.<o:p></o:p></span></b></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:11.0pt;
font-family:"Calibri","sans-serif";color:#E36C0A'>Keith<o:p></o:p></span></b></p>
</div>
</div>
</body>
</html>