<br><font size=2 face="sans-serif">Right. I think all of the "suggested"
implementations do not need a special allfenceall interface because they
are essentially variations on the "fence or allfence, then barrier"
idea. To justify a new interface there must be some way that some implementation
can use hardware-specific optimizations that are not exposed through the
existing pt2pt and collective primitives. I'm just say that experience
tells me we should be able to do something interesting in this area on
BG/P. I agree that is not a very compelling argument, but I haven't
the time to come up with a full bgp design. :) Maybe Jeff, myself,
and Brian can hash something out that is a little more real.</font>
<br>
<br><font size=2 face="sans-serif">I probably didn't wordsmith this very
well, but I mean to say is that some hardware features are only available
to a platform at the very lowest levels of the software stack. For example,
on BG/P we have the raw DMA SPI which represent 100% of the hardware features,
but the public C interface for DCMF doesn't provide all these details to
its users (MPICH2 DCMFd "glue" and ARMCI). For DCMF we have the
luxury of adding interfaces as the needs arise, and we would design a new
interface so that the guts of DCMF could use as many SPI tricks as possible
to implement the semantics of the interface for the user, as opposed to
reusing existing DCMF C API functions to implement the semantics o the
interface.</font>
<br>
<br>
<br><font size=2 face="sans-serif">Deterministic routing performance ..
</font>
<br>
<br><font size=2 face="sans-serif">Well, it depends on what you are trying
to do. :) For bisection bandwidth you'll definitely need dynamic
routing to avoid network hotspots. However, deterministic routing allows
us to play games to eliminate, avoid, or limit acks or active remote participation.
The problem with counter arrays or any data structure that grows with the
size of the job is you just run out of memory. This shows up on BG/P in
virtual node mode at scale where every additional 32 bits of a data structure
will result in an additional 1MB of memory usage - and there is only 256
MB available per rank in vnm! </font>
<br>
<br><font size=2 face="sans-serif">We use deterministic routing to provide
a "remote completion" callback event for DCMF_Put that only uses
the dma hardware - no cores - so that the put is truly one sided. The put
packets are all deterministically routed and then followed by a deterministically
routed "remote get" packet. The target dma will process the remote
get packet only after it has processed all of the put packets. When the
primitive get operation completes on the origin we know that all of the
previous put packets have been pulled from the network and written to memory
on the target node.</font>
<br>
<br><font size=2 face="sans-serif">I could go on and on, but don't want
to pollute this thread any further. :)</font>
<br>
<br><font size=2 face="sans-serif">Michael Blocksome<br>
Blue Gene Messaging<br>
Advanced Systems Software Development<br>
blocksom@us.ibm.com<br>
</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">From:</font>
<td><font size=1 face="sans-serif">"Underwood, Keith D" <keith.d.underwood@intel.com></font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">To:</font>
<td><font size=1 face="sans-serif">"MPI 3.0 Remote Memory Access working
group" <mpi3-rma@lists.mpi-forum.org></font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Date:</font>
<td><font size=1 face="sans-serif">05/24/2010 02:09 PM</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Subject:</font>
<td><font size=1 face="sans-serif">Re: [Mpi3-rma] RMA proposal 1 update</font></table>
<br>
<hr noshade>
<br>
<br>
<br><font size=2 color=#004080 face="Calibri"> </font>
<br><font size=2 face="Arial">I think the discussion of the three possible
implementations below misses the mark a bit. Any low level messaging
layer (DCMF, LAPI, even "MPI") that can optimize the allreduce
and barrier operations could implement the allfenceall operation as described
below. Even though DCMF - which what MPICH2 on BG/P is based on -
uses the BG/P collective network, it still is *only* a fast allreduce that
has benefits over other pt2pt allreduce algorithms and not very interesting
when considering how an allfenceall would be implemented. </font><font size=3 face="Times New Roman"><br>
</font><font size=2 face="Arial"><br>
If we look at any exascale goal(s), as Jeff mentioned earlier, we'll see
that any time an application or middleware needs to track counters we won't
get the scaling required (assuming the allfenceall is flexible and not
restricted to only "comm_world" operations). What is needed is
an interface that allows low level software to take advantage of any hardware
or network features that would enable a "counter-less" allfenceall
implementation. On BG/P, this would probably take advantage of features
such as the torus dimensions of the network, the (configurable) deterministic
packet routing, the "deposit bit" in the packet header, etc.
I'm not saying I have any specific design in mind, just that experience
tells me that we should be able to do something cool in this space supported
by the BG hardware which would eliminate the need for software accounting.</font><font size=3 face="Times New Roman">
<br>
</font>
<br><font size=2 color=#e26200 face="Calibri"><b>The forum was relatively
clear in requiring an implementation for new calls. In theory, that
implementation kind of has to fit with the motivation for the call. Saying
“this would give performance advantages if we did something special, but
this implementation doesn’t give those advantages” isn’t particularly
compelling. The implementations you mention from earlier in the
thread could colloquially be called “the way it is done now”, “the way
IBM suggested”, and “the way Keith suggested that may be completely ridiculous
”. Better implementations would certainly be interesting to discuss.</b></font>
<br><font size=2 color=#e26200 face="Calibri"><b>BTW, in many of the RMA
discussions, it has been asserted that using the deterministic packet routing
is a terrible performance hit. Is that not an issue?</b></font>
<br><font size=2 face="Arial"><br>
If there was an allfenceall interface, then low level message could do
whatever it takes to get performance and scaling. Without the allfenceall
interface middleware is forced to use one of these platform independent
algorithms (allreduce, barrier, etc) and accept any inherent scaling and
performance restrictions.</font><font size=3 face="Times New Roman"> </font>
<br><font size=2 color=#e26200 face="Calibri"><b>I don’t understand this
comment. Without the allfenceall, I’m not sure how you would call
a collective to do a fenceall at a given node (fenceall would complete
all outstanding requests from a given rank). I’m also not
sure why the difference between an allfenceall and a fenceall makes a difference
between platform specific and platform independent approaches.</b></font>
<br><font size=2 color=#e26200 face="Calibri"><b>Keith</b></font><tt><font size=2>_______________________________________________<br>
mpi3-rma mailing list<br>
mpi3-rma@lists.mpi-forum.org<br>
</font></tt><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma"><tt><font size=2>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma</font></tt></a><tt><font size=2><br>
</font></tt>
<br>
<br>