[Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement

[Bronis R. de Supinski]

Jeff:

Re:

[snip]

> > A properly designed subsetting approach will NOT try to
> > partition the profiling interface into its own subset.
> > Instead, the interface must be partitioned into different
> > subsets, with each PMPI function in the same subset as the
> > corresponding MPI function.
>
> I'm not sure that I understand your rationale here (perhaps we can
> discuss more in Chicago?).

Sure, we can discuss it in Chicago. However, The rationale is
simple enough. The user of a subset wants to be able to profile
the functionality in a given subset.

> In my view of the world, at least one way to effect subsetting could
> be something that is dynamically loaded (or not) at run-time.  Hence,
> you could have PMPI or not, depending on options given to mpirun (for
> example).  I'm obviously waving my hands a bit here, but that's the
> premise I was assuming (details TBD, of course).

I have no objection to your premise. However, this proposal
is specifically designed to prevent dynamic expansion of the
feature set to include the PMPI calls (or at least to doing
so effectively since there are no MPI calls between which
the tool could interpose). In fact, the ability to change
the subset dynamically is beyond what I was requiring (but
is desirable). All I stated was that the expansion should
not require recompilation in order to include the profiling
interface for what is included in the subset.

If you (or anyone else) can design a mechanism that allows
using the macros that Rainer wants to enable without precluding
that interposition dynamically then I offer no opposition to
the idea. However, I would also suggest that such a mechanism
does not need any change to the standard; it is already compliant.

>  From your later mail:
>
> > Also, while I see your point that only a small set of
> > stuff needs to be placed into mpi.h, that does not
> > address the real question: What is the actual application
> > level benefit? Why do I care if I can get MPI_Comm_size
> > or MPI_Comm_rank inlined? Good programming practice
> > does not call them frequently enough for any performance
> > gain to matter. Alternatively, suppose I can squash some
> > stuff out of the MPI_Send call path. Will it really matter
> > to real applications? It seems unlikely, particularly
> > relative to the cost of lost flexibility for understanding
> > communication behavior and performance.
>
>
> Rainer's proposal came from joint discussions with NEC where (correct
> me if I'm wrong, Hubert) select functions *are* [optionally?] inlined
> in mpi.h because the cost of function calls on their platform is so
> high.

While one can argue whether the standard should avoid precluding
the exploitation of as yet unbuilt clever hardware mechanisms,
it seems pretty clear that it is not the job of the standard to
accommodate poorly designed hardware. If the cost of function
calls is so excessively high that ONE prevents effective use of
the hardware then the vendor needs to reconsider their design.

In any event, it appears that you are stating that a solution was
found: offer a non-compliant alternative implementation. This is
a far more appropriate solution in this case. Since I don't believe
Rainer was proposing that the PMPI interface should be optional
(just somehow not included in a base subset), the implementor
still must support it. That means no implementation effort is
saved by saying they could have a compliant implementation that
does not always support the interface (but always offers the
option to include that support). The implementor is always free
to offer non-compliant alternatives; it is up the users whether
they want to build with it.

> Additionally, I'm always wary of the term "real applications" -- what
> a "real" application is to a US DOE lab is very different than what a
> "real application" is to, say, someone in the oil and gas
> industry.  :-)  The scale difference alone is enough to make the
> latency gains by inlining select MPI functions important (meaning: non-
> US-DOE-labs tend to run at [much] smaller scale; fabric latency may be
> constrained to a single switch, and therefore *can* see benefit from
> inlining).
>
> As I understand Rainer's proposal, the inlining aspects also
> predicated on apps that enter production and are never changed.
> Perhaps this is not a pattern used much at DOE labs, but it is a
> pattern I see with many customers.

I'm not going to get into a pissing match with you over what
applications do what. You seem to almost be using "US DOE lab"
pejoratively. What I can say without the slightest hesitation
is that recompilation is often a significant barrier to tool
acceptance for application users and those users are NOT
limited to one site or even one type of site (yes, I know users
who are not at "DOE labs").

My opposition to changes that would force tools to require
recompilation is borne from experience.

Bronis