<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.3354" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left>
<P><FONT face=Arial color=#0000ff size=2>Hi,</FONT></P>
<P><FONT face=Arial><FONT color=#0000ff><FONT size=2><SPAN
class=586134223-10122008>Thanks. </SPAN>A couple of <SPAN
class=586134223-10122008>closing </SPAN>arguments<SPAN class=586134223-10122008>
against both proposals:</SPAN></FONT></FONT></FONT></P>
<P><FONT face=Arial color=#0000ff size=2>8) The IMPI standard (Interoperable
MPI, see </FONT><A href="http://impi.nist.gov/"><U><FONT face=Arial
color=#0000ff size=2>http://impi.nist.gov/</FONT></U></A><FONT face=Arial
color=#0000ff size=2>) requires all data to be put on the wire in the external32
representation. The most natural way to do this for a little-endian platform is
to transform the data in place<SPAN class=586134223-10122008> on the
sender</SPAN>. The proposed change<SPAN class=586134223-10122008>s</SPAN> will
effectively disadvantage little-endian platforms in the area of MPI
interoperability.</FONT></P>
<P><FONT face=Arial color=#0000ff size=2>9)<SPAN class=586134223-10122008> To
implement external32 representation for the f</SPAN>ile I/O<SPAN
class=586134223-10122008>, </SPAN>one may want to do the in-place
conversion <SPAN class=586134223-10122008>on the source process </SPAN>to
implement the external32 datarep. <SPAN class=586134223-10122008>One may
also want to map the file I/O upon pt2pt and collective communication.
</SPAN>Again, the proposed change<SPAN class=586134223-10122008>s</SPAN> will
hit the little-endian platforms.</FONT></P>
<P><SPAN class=586134223-10122008><FONT face=Arial color=#0000ff size=2>Little
endian is used a lot in the HPC now and will be used a lot in the future I
hope.</FONT></SPAN></P>
<P><SPAN class=586134223-10122008><FONT face=Arial color=#0000ff size=2>Does the
fate of a </FONT></SPAN><SPAN class=586134223-10122008><FONT face=Arial
color=#0000ff size=2>couple of incorrect programs, yet to be identified,
outweighs specific concerns expressed in this trail, and the potential
disadvantage to the little endian HPC platforms in particular? Let the Forum
decide.</FONT></SPAN></P>
<P><FONT face=Arial color=#0000ff size=2>Best regards.</FONT></P>
<P><FONT face=Arial color=#0000ff size=2>Alexander</FONT></P></DIV>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> mpi-22-bounces@lists.mpi-forum.org
[mailto:mpi-22-bounces@lists.mpi-forum.org] <B>On Behalf Of </B>Richard
Treumann<BR><B>Sent:</B> Wednesday, December 10, 2008 7:45 PM<BR><B>To:</B> MPI
2.2<BR><B>Subject:</B> Re: [Mpi-22] please review - Send Buffer Access (ticket
#45)<BR></FONT><BR></DIV>
<DIV></DIV>
<P>Brian has stated the issue related to the send buffer access proposal very
well. <BR><BR>We have seen some arguments that focus on how hard the rule is for
users, how counter intuitive it is and how often it is violated either naively
or knowingly. These are important issues but not the only issues.<BR><BR>We have
seen arguments about the ways optimization options could be lost if the rule is
changed. These are important issues but not the only issues.<BR><BR>As Brian
states: there is a cost benefit analysis here, not a perfect or even obvious
right answer.<BR><BR>Early in the debate I raised the question about whether we
could be risking optimizations and reminded people that it really does matter.
That was at a stage when I perceived a lot of passion based on user centered
arguments and almost no discussion of the other side. The tone I perceived was
basically "the rationale in the original standard seems to be moot so all that
remains is the user centered argument". I felt the counter argument at least
needed to be examined. That has happened.<BR><BR>I think we have now reached a
point where both sides of the debate have been aired. People on each side have
heard and considered the counter arguments. I think most forum members would
agree that we cannot be absolutely certain whether keeping the send buffer
restriction would ever prove to be valuable. I still think there is some risk in
removing the restriction but I also see substantial value in removing
it.<BR><BR><B>My take is that as a cost/benefit decision we should remove the
restriction on send buffer access.</B><BR><BR><B>I also think putting the
compiler attribute "const" on a send buffer parameter should be voted down.
</B><BR><BR>The formal argument to a send seen by the compiler may or may not
correspond to the buffer. The datatype offsets play a role too. This most
obvious case is when MPI_BOTTOM is the send argument but there are other
examples. <BR>MPI_Send(&(array[0]) ...) <BR>MPI_Send(array,....)
<BR>MPI_Send(&var, ....) <BR>MPI_Send(MPI_BOTTOM, ......) <BR>are all valid
ways of sending the content of array[10] when combined with a suitable datatype.
<BR>(for the "var" example we would need a datatype that set MPI_LB at addr(var)
and used ( addr(array[10]-addr(var) ) as a displacement. Weird but
valid.)<BR><BR>The complier optimizations that can come from adding "const" are
probably small. The const attribute is semantically inaccurate if we consider
MPI_BOTTOM to represent the entire memory array. Every subroutine call alters
some portions of memory. I presume the compiler just recognizes that it has no
idea what range MPI_BOTTOM represents and ignores the "const".<BR><BR>Dick
<BR><BR>Dick Treumann - MPI Team <BR>IBM Systems & Technology Group<BR>Dept
X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601<BR>Tele (845)
433-7846 Fax (845) 433-8363<BR><BR><BR><TT>mpi-22-bounces@lists.mpi-forum.org
wrote on 12/10/2008 12:36:05 AM:<BR><BR>> [image removed] </TT><BR><TT>>
<BR>> Re: [Mpi-22] please review - Send Buffer Access (ticket
#45)</TT><BR><TT>> <BR>> Barrett, Brian W </TT><BR><TT>> <BR>>
to:</TT><BR><TT>> <BR>> MPI 2.2</TT><BR><TT>> <BR>> 12/10/2008 12:45
AM</TT><BR><TT>> <BR>> Sent by:</TT><BR><TT>> <BR>>
mpi-22-bounces@lists.mpi-forum.org</TT><BR><TT>> <BR>> Please respond to
"MPI 2.2"</TT><BR><TT>> <BR>> Ok, now that I've gotten my user's point of
view off my chest, back to my<BR>> view as an MPI implementer. I've
tried to respond to all Alexander's<BR>> objections. If I missed one, I
apologize.<BR>> <BR>> > 1) The memory remapping scenario IO brought up
a couple of days ago<BR>> <BR>> Unless someone's done something fun with
operating system and architecture<BR>> design I'm not aware of, remapping has
one problem that always must be<BR>> considered... Send buffers are not
required to be (and frequently aren't)<BR>> page aligned or a multiple of
page size. Therefore, completely removing the<BR>> send buffer from the
user's process has the problem of also taking legal<BR>> addresses with it
(which would violate the standard). IBM's solution is<BR>>
elegant in that it allows remapping without removing from the sender's<BR>>
process space. Sandia has a solution called SMARTMAP that is both
not<BR>> patented and allows single copy transfers in shared memory
environments.<BR>> <BR>> Point number 2 was a procedural argument. I
believe others are in a better<BR>> position than I to comment on this.
My understanding, however, is that a<BR>> technical objection can cause
the vote to fail, but is not grounds on<BR>> preventing a vote (particularly
a second vote). If it were, we'd never get<BR>> anything done.<BR>>
<BR>> > 3) Imagine send buffers have to pinned in the memory. To avoid
<BR>> doing this too<BR>> > often, these registrations will normally be
cached. If more than <BR>> one send can<BR>> > be used for a buffer or,
for that matter, overlapping portions of the same<BR>> > buffer, say by
different threads, access to the lookup-and-pin <BR>> will have to be<BR>>
> made atomic. This will further complicate implementation and introduce
a<BR>> > potentially costly mutual exclusion primitive into the critical
path.<BR>> <BR>> The caching problem already exists. Consider a case
where a large send is<BR>> completed, then multiple small sends occur within
that base and bound after<BR>> the first is completed. This situation
is perfectly legal, happens in codes<BR>> in the wild, and must be dealt with
by MPI implementations. If that's not<BR>> enough, consider a case
where the buffer is part of an active Window (which<BR>> is legal, as long as
the buffers in use for communication don't overlap).<BR>> All these cases
certainly should be handled by an MPI today.<BR>> <BR>> > 4) I wonder
what a const modifier will do for a buffer identifies by<BR>> > MPI_BOTTOM
and/or a derived data type, possibly with holes in it. How will<BR>> >
this square up with the C language sequence association rules?<BR>> <BR>>
This sounds like an issue for the const proposal, which is different
from<BR>> the send buffer access proposal. I'm not sure I have enough
data to form an<BR>> opinion on the const proposal, but I'm fairly sure we
can discuss the send<BR>> buffer access proposal without considering this
issue.<BR>> <BR>> > 5) Note also if both #45 and #46 will be
introduced, there will beno way to<BR>> > retract this, even with the help
of the MPI_INIT_ASSERTED, should we later<BR>> > decide to introduce
assertion like MPI_NO_SEND_BUFFER_READ_ACCESS.The const<BR>> > modifier
from #46 will make that syntactically useless.<BR>> <BR>> If both are
passed, that might be true. It could be argued the const<BR>> proposal
depends on the access proposal. However, it can not be rationally<BR>>
argued that the access proposal in any way depends upon the const
proposal.<BR>> <BR>> The send buffer access proposal can certainly be
passed and an assert added<BR>> later (at whatever point the init_assert
proposal is integrated into the<BR>> standard) that allows MPI
implementations to modify the send buffer.<BR>> <BR>> You raise a good
point about the const proposal. But it has absolutely no<BR>> bearing
on the send buffer access proposal.<BR>> <BR>> > 6) Finally, what will
happen in the Fortran interface? With the<BR>> > copy-in/copy-out possibly
happening on the MPI subroutine boundaryfor array<BR>> > sections? If more
than one send is allowed, the application can <BR>> pretty easily<BR>>
> exhaust any virtual memory with a couple of long enough vectors.<BR>>
<BR>> How does that change from today? Today users send multiple
buffers at the<BR>> same time, and seem to cope with memory exhaustion issues
just fine. So<BR>> soon they might be able to remove the data copy
they've had to make at the<BR>> user level to work around the MPI access
restriction, so there's actually<BR>> less virtual memory in use. Seems
like a win to me.<BR>> <BR>> > 7) In-place compression and/or
encryption of the messages. Compression in<BR>> > particular can work
wonders on monotonous messages, and cost less time in<BR>> > total than
the transmission of so many giga-zeroes, for example. <BR>> Again,
having<BR>> > send buffer access allowed and const modifier attached will
kill this huge<BR>> > optimization opportunity. Too bad.<BR>> <BR>>
While I hope you're joking about the giga-zeros, you do raise a valid<BR>>
concern, in that there are a number of optimizations regarding
compression,<BR>> encryption, and endian-swapping that may be eliminated by
this proposal. On<BR>> the flip side, as I argued in a previous e-mail,
the user gains quite a bit<BR>> in usability. We have to balance these
two factors. Since users know where<BR>> my office is, I tend to lean
towards making their lives easier, particularly<BR>> when it doesn't cause
extra work for me. But I already sent an e-mail on<BR>> that
point...<BR>> <BR>> Our experience with Open MPI was that the potential
for performance in other<BR>> parts of the MPI (collectives, etc.) far
outweighed any send-side tricks we<BR>> could think of (and you haven't
brought up any we didn't think of). So if<BR>> we wanted to do
compression or encryption, it would be done with send-side<BR>> bounce
buffers. Since a software pipeline would practically be required
to<BR>> get good performance, the bounce buffer would not have to scale with
the<BR>> size of the communication buffer but instead with the properties of
the<BR>> network pipeline. Of course, my opinion would be that it would
be much<BR>> simpler and much higher performance to support compression or
encryption as<BR>> part of the NIC as the data is streamed to the network.
Otherwise, you're<BR>> burning memory bandwidth doing the extra copy
(even in the modify the send<BR>> buffer case), and memory bandwidth is a
precious resource for HPC<BR>> applications.<BR>> <BR>> One other point
to consider. If I was a user, I'd expect that my one-sided<BR>> traffic
also be compressed, encrypted, or endian-swapped. The standard<BR>>
already requires multiple accesses be legal for one-sided communication.
So<BR>> you're going to have a situation where some communication can
use a<BR>> send-modify implementation and some can not. I'm not
familiar with how<BR>> Intel's MPI is architected, but Open MPI is
architected such that decisions<BR>> such as compression, encryption, and
endian-swapping would be made at a low<BR>> enough level that the code path
is the same whether the message is a<BR>> point-to-point send or a one-sided
put. Since that's some of the most<BR>> complicated code in Open MPI, I
can't foresee adding a second code path just<BR>> to get a (dubious)
performance benefit.<BR>> <BR>> <BR>> Brian<BR>> <BR>> --<BR>>
Brian W. Barrett<BR>> Dept. 1422: Scalable Computer
Architectures<BR>> Sandia National Laboratories<BR>> <BR>>
<BR>> <BR>> _______________________________________________<BR>> mpi-22
mailing list<BR>> mpi-22@lists.mpi-forum.org<BR>> <A
href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22</A><BR></TT></P><pre>---------------------------------------------------------------------
Intel GmbH
Dornacher Strasse 1
85622 Feldkirchen/Muenchen Germany
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer
Registergericht: Muenchen HRB 47456 Ust.-IdNr.
VAT Registration No.: DE129385895
Citibank Frankfurt (BLZ 502 109 00) 600119052
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
</pre></BODY></HTML>