<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Sorry, but you lost me at “we could just do an allreduce to look
at counts”. Could you go into a bit more detail? If you have received counts
from all ranks at all ranks (um, that doesn’t seem scalable), then it would seem
that an allfenceall() would require an Alltoall() to figure out if everybody
was safe. I don’t see how an allreduce would do the job. But, I’ll admit
that I don’t know really DCMF or BG network interface architecture or… So, I
could just be missing something here.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Thanks,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Keith<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
mpi3-rma-bounces@lists.mpi-forum.org
[mailto:mpi3-rma-bounces@lists.mpi-forum.org] <b>On Behalf Of </b>Brian Smith<br>
<b>Sent:</b> Tuesday, May 18, 2010 4:57 AM<br>
<b>To:</b> MPI 3.0 Remote Memory Access working group<br>
<b>Cc:</b> MPI 3.0 Remote Memory Access working group;
mpi3-rma-bounces@lists.mpi-forum.org<br>
<b>Subject:</b> Re: [Mpi3-rma] RMA proposal 1 update<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
<tt><span style='font-size:10.0pt'>Sorry for the late response.... </span></tt><br>
<tt><span style='font-size:10.0pt'>On BGP, DCMF Put/Get doesn't do any
accounting and DCMF doesn't actually have a fence operation. There is no
hardware to determine when a put/get has completed either. We need to send a
get along the same (deterministically routed) path to "flush" any
messages out to claim we are synchronized.</span></tt><span style='font-size:
10.0pt;font-family:"Courier New"'><br>
<br>
<tt>When we implemented ARMCI, we introduced accounting in our "glue"
on top of DCMF because of the ARMCI_Fence() operation. There are similar
concerns in the MPI one-sided "glue".</tt><br>
<br>
<tt>Going forward, we need to figure out how we'd implement the new MPI RMA
operations and determine if there would be accounting required. If there would
be (and I'm thinking there would), then an allfenceall in MPI would be easy
enough to do and would provide a significant benefit on BG. We could just do an
allreduce to look at counts. If the standard procedure is fenceall()+barrier(),
I could do that much better as an allfenceall call.</tt></span> <br>
<br>
<tt><span style='font-size:10.0pt'>On platforms that have some sort of native
accounting, this allfenceall would only be the overhead of a barrier. So I
think an allfenceall has significant value to the middleware more than DCMF and
therefore would strongly encourage it in MPI, especially given the use-cases we
heard from Jeff H. at the forum meeting.</span></tt> <br>
<span style='font-size:10.0pt;font-family:"Courier New"'><br>
<tt>This scenario is the same in our next super-secret product offering
everyone knows about but I don't know if *I* can mention.</tt></span> <br>
<br>
<span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><br>
Brian Smith (smithbr@us.ibm.com)<br>
BlueGene MPI Development/<br>
Communications Team Lead<br>
IBM Rochester<br>
Phone: 507 253 4717</span> <br>
<br>
<o:p></o:p></p>
<table class=MsoNormalTable border=0 cellpadding=0 width="100%"
style='width:100.0%'>
<tr>
<td valign=top style='padding:.75pt .75pt .75pt .75pt'>
<p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif";
color:#5F5F5F'>From:</span> <o:p></o:p></p>
</td>
<td valign=top style='padding:.75pt .75pt .75pt .75pt'>
<p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>"Underwood,
Keith D" <keith.d.underwood@intel.com></span> <o:p></o:p></p>
</td>
</tr>
<tr>
<td valign=top style='padding:.75pt .75pt .75pt .75pt'>
<p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif";
color:#5F5F5F'>To:</span> <o:p></o:p></p>
</td>
<td valign=top style='padding:.75pt .75pt .75pt .75pt'>
<p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>"MPI
3.0 Remote Memory Access working group"
<mpi3-rma@lists.mpi-forum.org></span> <o:p></o:p></p>
</td>
</tr>
<tr>
<td valign=top style='padding:.75pt .75pt .75pt .75pt'>
<p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif";
color:#5F5F5F'>Date:</span> <o:p></o:p></p>
</td>
<td valign=top style='padding:.75pt .75pt .75pt .75pt'>
<p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>05/16/2010
09:33 PM</span> <o:p></o:p></p>
</td>
</tr>
<tr>
<td valign=top style='padding:.75pt .75pt .75pt .75pt'>
<p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif";
color:#5F5F5F'>Subject:</span> <o:p></o:p></p>
</td>
<td valign=top style='padding:.75pt .75pt .75pt .75pt'>
<p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>Re:
[Mpi3-rma] RMA proposal 1 update</span> <o:p></o:p></p>
</td>
</tr>
<tr>
<td valign=top style='padding:.75pt .75pt .75pt .75pt'>
<p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif";
color:#5F5F5F'>Sent by:</span> <o:p></o:p></p>
</td>
<td valign=top style='padding:.75pt .75pt .75pt .75pt'>
<p class=MsoNormal><span style='font-size:7.5pt;font-family:"Arial","sans-serif"'>mpi3-rma-bounces@lists.mpi-forum.org</span><o:p></o:p></p>
</td>
</tr>
</table>
<p class=MsoNormal><o:p> </o:p></p>
<div class=MsoNormal align=center style='text-align:center'>
<hr size=2 width="100%" noshade style='color:#ACA899' align=center>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
<br>
<br>
<tt><span style='font-size:10.0pt'>Before doing that, can someone sketch out
the platform/API and the implementation that makes that more efficient?
There is no gain for Portals (3 or 4). There is no gain for
anything that supports Cray SHMEM reasonably well (shmem_quiet() is
approximately the same semantics as MPI_flush_all). Hrm, you can probably
say the same thing about anything that supports UPC well - a strict access is
basically a MPI_flush_all(); MPI_Put(); MPI_flush_all();... Also, I thought
somebody said that IB gave you a notification of remote completion...</span></tt><span
style='font-size:10.0pt;font-family:"Courier New"'><br>
<br>
<tt>The question then turns to the "other networks". If you
can't figure out remote completion, then the collective is going to be pretty
heavy, right?</tt><br>
<br>
<tt>Keith</tt><br>
<br>
<tt>> -----Original Message-----</tt><br>
<tt>> From: mpi3-rma-bounces@lists.mpi-forum.org [</tt></span><a
href="mailto:mpi3-rma-"><tt><span style='font-size:10.0pt'>mailto:mpi3-rma-</span></tt></a><span
style='font-size:10.0pt;font-family:"Courier New"'><br>
<tt>> bounces@lists.mpi-forum.org] On Behalf Of Jeff Hammond</tt><br>
<tt>> Sent: Sunday, May 16, 2010 7:27 PM</tt><br>
<tt>> To: MPI 3.0 Remote Memory Access working group</tt><br>
<tt>> Subject: Re: [Mpi3-rma] RMA proposal 1 update</tt><br>
<tt>> </tt><br>
<tt>> Tortsten,</tt><br>
<tt>> </tt><br>
<tt>> There seemed to be decent agreement on adding MPI_Win_all_flush_all</tt><br>
<tt>> (equivalent to MPI_Win_flush_all called from every rank in the</tt><br>
<tt>> communicator associated with the window) since this function can be</tt><br>
<tt>> implemented far more efficiently as a collective than the equivalent</tt><br>
<tt>> point-wise function calls.</tt><br>
<tt>> </tt><br>
<tt>> Is there a problem with adding this to your proposal?</tt><br>
<tt>> </tt><br>
<tt>> Jeff</tt><br>
<tt>> </tt><br>
<tt>> On Sun, May 16, 2010 at 12:48 AM, Torsten Hoefler
<htor@illinois.edu></tt><br>
<tt>> wrote:</tt><br>
<tt>> > Hello all,</tt><br>
<tt>> ></tt><br>
<tt>> > After the discussions at the last Forum I updated the group's
first</tt><br>
<tt>> > proposal.</tt><br>
<tt>> ></tt><br>
<tt>> > The proposal (one-side-2.pdf) is attached to the wiki page</tt><br>
<tt>> > </tt></span><a
href="https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage"><tt><span
style='font-size:10.0pt'>https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage</span></tt></a><span
style='font-size:10.0pt;font-family:"Courier New"'><br>
<tt>> ></tt><br>
<tt>> > The changes with regards to the last version are:</tt><br>
<tt>> ></tt><br>
<tt>> > 1) added MPI_NOOP to MPI_Get_accumulate and MPI_Accumulate_get</tt><br>
<tt>> ></tt><br>
<tt>> > 2) (re)added MPI_Win_flush and MPI_Win_flush_all to passive
target</tt><br>
<tt>> mode</tt><br>
<tt>> ></tt><br>
<tt>> > Some remarks:</tt><br>
<tt>> ></tt><br>
<tt>> > 1) We didn't straw-vote on MPI_Accumulate_get, so this function
might</tt><br>
<tt>> > go. The removal would be very clean.</tt><br>
<tt>> ></tt><br>
<tt>> > 2) Should we allow MPI_NOOP in MPI_Accumulate (this does not make</tt><br>
<tt>> sense</tt><br>
<tt>> > and is incorrect in my current proposal)</tt><br>
<tt>> ></tt><br>
<tt>> > 3) Should we allow MPI_REPLACE in</tt><br>
<tt>> MPI_Get_accumulate/MPI_Accumulate_get?</tt><br>
<tt>> > (this would make sense and is allowed in the current
proposal but</tt><br>
<tt>> we</tt><br>
<tt>> > didn't talk about it in the group)</tt><br>
<tt>> ></tt><br>
<tt>> ></tt><br>
<tt>> > All the Best,</tt><br>
<tt>> > Torsten</tt><br>
<tt>> ></tt><br>
<tt>> > --</tt><br>
<tt>> > bash$ :(){ :|:&};: --------------------- </tt></span><a
href="http://www.unixer.de/"><tt><span style='font-size:10.0pt'>http://www.unixer.de/</span></tt></a><tt><span
style='font-size:10.0pt'> -----</span></tt><span style='font-size:10.0pt;
font-family:"Courier New"'><br>
<tt>> > Torsten Hoefler | Research Associate</tt><br>
<tt>> > Blue Waters Directorate | University of Illinois</tt><br>
<tt>> > 1205 W Clark Street | Urbana, IL, 61801</tt><br>
<tt>> > NCSA Building | +01 (217) 244-7736</tt><br>
<tt>> > _______________________________________________</tt><br>
<tt>> > mpi3-rma mailing list</tt><br>
<tt>> > mpi3-rma@lists.mpi-forum.org</tt><br>
<tt>> > </tt></span><a
href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma"><tt><span
style='font-size:10.0pt'>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma</span></tt></a><span
style='font-size:10.0pt;font-family:"Courier New"'><br>
<tt>> ></tt><br>
<tt>> </tt><br>
<tt>> </tt><br>
<tt>> </tt><br>
<tt>> --</tt><br>
<tt>> Jeff Hammond</tt><br>
<tt>> Argonne Leadership Computing Facility</tt><br>
<tt>> jhammond@mcs.anl.gov / (630) 252-5381</tt><br>
<tt>> </tt></span><a href="http://www.linkedin.com/in/jeffhammond"><tt><span
style='font-size:10.0pt'>http://www.linkedin.com/in/jeffhammond</span></tt></a><span
style='font-size:10.0pt;font-family:"Courier New"'><br>
<tt>> </tt><br>
<tt>> _______________________________________________</tt><br>
<tt>> mpi3-rma mailing list</tt><br>
<tt>> mpi3-rma@lists.mpi-forum.org</tt><br>
<tt>> </tt></span><a
href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma"><tt><span
style='font-size:10.0pt'>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma</span></tt></a><span
style='font-size:10.0pt;font-family:"Courier New"'><br>
<br>
<tt>_______________________________________________</tt><br>
<tt>mpi3-rma mailing list</tt><br>
<tt>mpi3-rma@lists.mpi-forum.org</tt><br>
</span><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma"><tt><span
style='font-size:10.0pt'>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma</span></tt></a><span
style='font-size:10.0pt;font-family:"Courier New"'><br>
<br>
</span><o:p></o:p></p>
</div>
</div>
</body>
</html>