<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Thanks Dick,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I think that you have somewhat wrong perception regarding the C
const keyword. The only promises that the C lang is making about passing
parameter as ‘const’ is that the function will not change the
content of <u>ANY</u> memory location through <u>THAT</u> pointer. As long as
you are not casting away constness in your function you are good and following
the const rules. You are allowed to change the content of that same
memory through a different pointer that is not const.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>All the examples that you have below are 100% valid with the
const keyword.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Thanks,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>.Erez<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
mpi-22-bounces@lists.mpi-forum.org [mailto:mpi-22-bounces@lists.mpi-forum.org] <b>On
Behalf Of </b>Richard Treumann<br>
<b>Sent:</b> Wednesday, December 10, 2008 10:45 AM<br>
<b>To:</b> MPI 2.2<br>
<b>Subject:</b> Re: [Mpi-22] please review - Send Buffer Access (ticket #45)<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p>Brian has stated the issue related to the send buffer access proposal very
well. <br>
<br>
We have seen some arguments that focus on how hard the rule is for users, how
counter intuitive it is and how often it is violated either naively or
knowingly. These are important issues but not the only issues.<br>
<br>
We have seen arguments about the ways optimization options could be lost if the
rule is changed. These are important issues but not the only issues.<br>
<br>
As Brian states: there is a cost benefit analysis here, not a perfect or even
obvious right answer.<br>
<br>
Early in the debate I raised the question about whether we could be risking
optimizations and reminded people that it really does matter. That was at a
stage when I perceived a lot of passion based on user centered arguments and
almost no discussion of the other side. The tone I perceived was basically "the
rationale in the original standard seems to be moot so all that remains is the
user centered argument". I felt the counter argument at least needed to be
examined. That has happened.<br>
<br>
I think we have now reached a point where both sides of the debate have been
aired. People on each side have heard and considered the counter arguments. I
think most forum members would agree that we cannot be absolutely certain
whether keeping the send buffer restriction would ever prove to be valuable. I
still think there is some risk in removing the restriction but I also see
substantial value in removing it.<br>
<br>
<b>My take is that as a cost/benefit decision we should remove the restriction
on send buffer access.</b><br>
<br>
<b>I also think putting the compiler attribute "const" on a send buffer
parameter should be voted down. </b><br>
<br>
The formal argument to a send seen by the compiler may or may not correspond to
the buffer. The datatype offsets play a role too. This most obvious case is
when MPI_BOTTOM is the send argument but there are other examples. <br>
MPI_Send(&(array[0]) ...) <br>
MPI_Send(array,....) <br>
MPI_Send(&var, ....) <br>
MPI_Send(MPI_BOTTOM, ......) <br>
are all valid ways of sending the content of array[10] when combined with a
suitable datatype. <br>
(for the "var" example we would need a datatype that set MPI_LB at
addr(var) and used ( addr(array[10]-addr(var) ) as a displacement. Weird but
valid.)<br>
<br>
The complier optimizations that can come from adding "const" are
probably small. The const attribute is semantically inaccurate if we consider
MPI_BOTTOM to represent the entire memory array. Every subroutine call alters
some portions of memory. I presume the compiler just recognizes that it has no
idea what range MPI_BOTTOM represents and ignores the "const".<br>
<br>
Dick <br>
<br>
Dick Treumann - MPI Team <br>
IBM Systems & Technology Group<br>
Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601<br>
Tele (845) 433-7846 Fax (845) 433-8363<br>
<br>
<br>
<tt><span style='font-size:10.0pt'>mpi-22-bounces@lists.mpi-forum.org wrote on
12/10/2008 12:36:05 AM:</span></tt><span style='font-size:10.0pt;font-family:
"Courier New"'><br>
<br>
<tt>> [image removed] </tt></span><br>
<tt><span style='font-size:10.0pt'>> </span></tt><span style='font-size:
10.0pt;font-family:"Courier New"'><br>
<tt>> Re: [Mpi-22] please review - Send Buffer Access (ticket #45)</tt></span><br>
<tt><span style='font-size:10.0pt'>> </span></tt><span style='font-size:
10.0pt;font-family:"Courier New"'><br>
<tt>> Barrett, Brian W </tt></span><br>
<tt><span style='font-size:10.0pt'>> </span></tt><span style='font-size:
10.0pt;font-family:"Courier New"'><br>
<tt>> to:</tt></span><br>
<tt><span style='font-size:10.0pt'>> </span></tt><span style='font-size:
10.0pt;font-family:"Courier New"'><br>
<tt>> MPI 2.2</tt></span><br>
<tt><span style='font-size:10.0pt'>> </span></tt><span style='font-size:
10.0pt;font-family:"Courier New"'><br>
<tt>> 12/10/2008 12:45 AM</tt></span><br>
<tt><span style='font-size:10.0pt'>> </span></tt><span style='font-size:
10.0pt;font-family:"Courier New"'><br>
<tt>> Sent by:</tt></span><br>
<tt><span style='font-size:10.0pt'>> </span></tt><span style='font-size:
10.0pt;font-family:"Courier New"'><br>
<tt>> mpi-22-bounces@lists.mpi-forum.org</tt></span><br>
<tt><span style='font-size:10.0pt'>> </span></tt><span style='font-size:
10.0pt;font-family:"Courier New"'><br>
<tt>> Please respond to "MPI 2.2"</tt></span><br>
<tt><span style='font-size:10.0pt'>> </span></tt><span style='font-size:
10.0pt;font-family:"Courier New"'><br>
<tt>> Ok, now that I've gotten my user's point of view off my chest, back to
my</tt><br>
<tt>> view as an MPI implementer. I've tried to respond to all
Alexander's</tt><br>
<tt>> objections. If I missed one, I apologize.</tt><br>
<tt>> </tt><br>
<tt>> > 1) The memory remapping scenario IO brought up a couple of days
ago</tt><br>
<tt>> </tt><br>
<tt>> Unless someone's done something fun with operating system and
architecture</tt><br>
<tt>> design I'm not aware of, remapping has one problem that always must be</tt><br>
<tt>> considered... Send buffers are not required to be (and
frequently aren't)</tt><br>
<tt>> page aligned or a multiple of page size. Therefore, completely
removing the</tt><br>
<tt>> send buffer from the user's process has the problem of also taking
legal</tt><br>
<tt>> addresses with it (which would violate the standard).
IBM's solution is</tt><br>
<tt>> elegant in that it allows remapping without removing from the sender's</tt><br>
<tt>> process space. Sandia has a solution called SMARTMAP that is
both not</tt><br>
<tt>> patented and allows single copy transfers in shared memory
environments.</tt><br>
<tt>> </tt><br>
<tt>> Point number 2 was a procedural argument. I believe others are
in a better</tt><br>
<tt>> position than I to comment on this. My understanding, however, is
that a</tt><br>
<tt>> technical objection can cause the vote to fail, but is not grounds on</tt><br>
<tt>> preventing a vote (particularly a second vote). If it were, we'd
never get</tt><br>
<tt>> anything done.</tt><br>
<tt>> </tt><br>
<tt>> > 3) Imagine send buffers have to pinned in the memory. To avoid </tt><br>
<tt>> doing this too</tt><br>
<tt>> > often, these registrations will normally be cached. If more than </tt><br>
<tt>> one send can</tt><br>
<tt>> > be used for a buffer or, for that matter, overlapping portions of
the same</tt><br>
<tt>> > buffer, say by different threads, access to the lookup-and-pin </tt><br>
<tt>> will have to be</tt><br>
<tt>> > made atomic. This will further complicate implementation and
introduce a</tt><br>
<tt>> > potentially costly mutual exclusion primitive into the critical
path.</tt><br>
<tt>> </tt><br>
<tt>> The caching problem already exists. Consider a case where a
large send is</tt><br>
<tt>> completed, then multiple small sends occur within that base and bound
after</tt><br>
<tt>> the first is completed. This situation is perfectly legal,
happens in codes</tt><br>
<tt>> in the wild, and must be dealt with by MPI implementations. If
that's not</tt><br>
<tt>> enough, consider a case where the buffer is part of an active Window
(which</tt><br>
<tt>> is legal, as long as the buffers in use for communication don't
overlap).</tt><br>
<tt>> All these cases certainly should be handled by an MPI today.</tt><br>
<tt>> </tt><br>
<tt>> > 4) I wonder what a const modifier will do for a buffer identifies
by</tt><br>
<tt>> > MPI_BOTTOM and/or a derived data type, possibly with holes in it.
How will</tt><br>
<tt>> > this square up with the C language sequence association rules?</tt><br>
<tt>> </tt><br>
<tt>> This sounds like an issue for the const proposal, which is different
from</tt><br>
<tt>> the send buffer access proposal. I'm not sure I have enough data
to form an</tt><br>
<tt>> opinion on the const proposal, but I'm fairly sure we can discuss the
send</tt><br>
<tt>> buffer access proposal without considering this issue.</tt><br>
<tt>> </tt><br>
<tt>> > 5) Note also if both #45 and #46 will be introduced, there will
beno way to</tt><br>
<tt>> > retract this, even with the help of the MPI_INIT_ASSERTED, should
we later</tt><br>
<tt>> > decide to introduce assertion like
MPI_NO_SEND_BUFFER_READ_ACCESS.The const</tt><br>
<tt>> > modifier from #46 will make that syntactically useless.</tt><br>
<tt>> </tt><br>
<tt>> If both are passed, that might be true. It could be argued the
const</tt><br>
<tt>> proposal depends on the access proposal. However, it can not be
rationally</tt><br>
<tt>> argued that the access proposal in any way depends upon the const
proposal.</tt><br>
<tt>> </tt><br>
<tt>> The send buffer access proposal can certainly be passed and an assert
added</tt><br>
<tt>> later (at whatever point the init_assert proposal is integrated into
the</tt><br>
<tt>> standard) that allows MPI implementations to modify the send buffer.</tt><br>
<tt>> </tt><br>
<tt>> You raise a good point about the const proposal. But it has
absolutely no</tt><br>
<tt>> bearing on the send buffer access proposal.</tt><br>
<tt>> </tt><br>
<tt>> > 6) Finally, what will happen in the Fortran interface? With the</tt><br>
<tt>> > copy-in/copy-out possibly happening on the MPI subroutine
boundaryfor array</tt><br>
<tt>> > sections? If more than one send is allowed, the application can </tt><br>
<tt>> pretty easily</tt><br>
<tt>> > exhaust any virtual memory with a couple of long enough vectors.</tt><br>
<tt>> </tt><br>
<tt>> How does that change from today? Today users send multiple
buffers at the</tt><br>
<tt>> same time, and seem to cope with memory exhaustion issues just fine.
So</tt><br>
<tt>> soon they might be able to remove the data copy they've had to make at
the</tt><br>
<tt>> user level to work around the MPI access restriction, so there's
actually</tt><br>
<tt>> less virtual memory in use. Seems like a win to me.</tt><br>
<tt>> </tt><br>
<tt>> > 7) In-place compression and/or encryption of the messages.
Compression in</tt><br>
<tt>> > particular can work wonders on monotonous messages, and cost less
time in</tt><br>
<tt>> > total than the transmission of so many giga-zeroes, for example. </tt><br>
<tt>> Again, having</tt><br>
<tt>> > send buffer access allowed and const modifier attached will kill
this huge</tt><br>
<tt>> > optimization opportunity. Too bad.</tt><br>
<tt>> </tt><br>
<tt>> While I hope you're joking about the giga-zeros, you do raise a valid</tt><br>
<tt>> concern, in that there are a number of optimizations regarding
compression,</tt><br>
<tt>> encryption, and endian-swapping that may be eliminated by this
proposal. On</tt><br>
<tt>> the flip side, as I argued in a previous e-mail, the user gains quite
a bit</tt><br>
<tt>> in usability. We have to balance these two factors. Since
users know where</tt><br>
<tt>> my office is, I tend to lean towards making their lives easier, particularly</tt><br>
<tt>> when it doesn't cause extra work for me. But I already sent an
e-mail on</tt><br>
<tt>> that point...</tt><br>
<tt>> </tt><br>
<tt>> Our experience with Open MPI was that the potential for performance in
other</tt><br>
<tt>> parts of the MPI (collectives, etc.) far outweighed any send-side tricks
we</tt><br>
<tt>> could think of (and you haven't brought up any we didn't think of).
So if</tt><br>
<tt>> we wanted to do compression or encryption, it would be done with
send-side</tt><br>
<tt>> bounce buffers. Since a software pipeline would practically be
required to</tt><br>
<tt>> get good performance, the bounce buffer would not have to scale with
the</tt><br>
<tt>> size of the communication buffer but instead with the properties of
the</tt><br>
<tt>> network pipeline. Of course, my opinion would be that it would
be much</tt><br>
<tt>> simpler and much higher performance to support compression or
encryption as</tt><br>
<tt>> part of the NIC as the data is streamed to the network.
Otherwise, you're</tt><br>
<tt>> burning memory bandwidth doing the extra copy (even in the modify the
send</tt><br>
<tt>> buffer case), and memory bandwidth is a precious resource for HPC</tt><br>
<tt>> applications.</tt><br>
<tt>> </tt><br>
<tt>> One other point to consider. If I was a user, I'd expect that my
one-sided</tt><br>
<tt>> traffic also be compressed, encrypted, or endian-swapped. The
standard</tt><br>
<tt>> already requires multiple accesses be legal for one-sided
communication. So</tt><br>
<tt>> you're going to have a situation where some communication can use a</tt><br>
<tt>> send-modify implementation and some can not. I'm not familiar
with how</tt><br>
<tt>> Intel's MPI is architected, but Open MPI is architected such that
decisions</tt><br>
<tt>> such as compression, encryption, and endian-swapping would be made at
a low</tt><br>
<tt>> enough level that the code path is the same whether the message is a</tt><br>
<tt>> point-to-point send or a one-sided put. Since that's some of the
most</tt><br>
<tt>> complicated code in Open MPI, I can't foresee adding a second code
path just</tt><br>
<tt>> to get a (dubious) performance benefit.</tt><br>
<tt>> </tt><br>
<tt>> </tt><br>
<tt>> Brian</tt><br>
<tt>> </tt><br>
<tt>> --</tt><br>
<tt>> Brian W. Barrett</tt><br>
<tt>> Dept. 1422: Scalable Computer Architectures</tt><br>
<tt>> Sandia National Laboratories</tt><br>
<tt>> </tt><br>
<tt>> </tt><br>
<tt>> </tt><br>
<tt>> _______________________________________________</tt><br>
<tt>> mpi-22 mailing list</tt><br>
<tt>> mpi-22@lists.mpi-forum.org</tt><br>
<tt>> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22</a></tt></span><o:p></o:p></p>
</div>
</body>
</html>