[Mpi-22] please review - Send Buffer Access (ticket #45)

[Barrett, Brian W]

For reasons I'm not sure I completely understand, the discussion on this
proposal has become focused on micro-optimizations for MPI implementations,
rather than the original desire to fix one of the many usability complaints
users have with the MPI standard.

I personally believe that the send buffer restriction places an onerous
burden on the developer of modern MPI applications.  While it's
straight-forward to ensure a buffer is not used multiple times for simple
ghost cell transfers in basic physics codes, it can be more challenging in
more advanced applications.  We already know of one application in the wild
that reuses send buffers (the Parallel BGL).  I'm certain that if we polled
application writers, we'd find even more which violate the restriction.
Even as an MPI implementer who knows what he's doing (in theory, anyway), I
find this restriction can be a difficult problem to work around.

Further, this is a difficult issue to detect.  Yes, a correctness-checking
library can track buffers in use.  But consider that buffer reuse is likely
to be quite transient, and correctness-checking libraries frequently have an
extraordinary performance penalty.  Normal development cycle testing likely
won't cause the issue to appear, since no current mainstream MPI
implementation enforces the rule and I'm doubtful that situation will
change.

Finally, from a user's perspective, it must look rather capricious of the
standards committee that a send buffer can't be reused but a one-sided
buffer can.  I certainly couldn't justify to a user why the standard has
such a contradiction.  Do we as a committee have the right to be capricious
on this issue?  Certainly.  Should we then wonder why there are so many
people running around preaching the evils of MPI?  Certainly not.

So in my opinion the question really should be framed not as "Is there any
optimization that such a change prohibits?", but "Does the usability of the
standard outweigh the loss of *potential* optimizations?".  I have to think
that eliminating a common source of confusion for users is worth losing the
ability to increase the performance of sending so many giga-zeros of data.

Brian


--
   Brian W. Barrett
   Dept. 1422: Scalable Computer Architectures
   Sandia National Laboratories