[Mpi-forum] large count support not as easy as people seem to have thought

[Hjelm, Nathan T]

+1 for large counts. I find it just a bit ridiculous that this was punted on by the forum but I wasn't around for the discussion. Is this in issue forced on C/C++ by fortran?

-Nathan
________________________________________
From: mpi-forum [mpi-forum-bounces at lists.mpi-forum.org] on behalf of Jeff Hammond [jeff.science at gmail.com]
Sent: Tuesday, May 06, 2014 11:19 AM
To: MPI Forum
Subject: [Mpi-forum] large count support not as easy as people seem to have     thought

As I noted on this list last Friday, I am working on a higher-level
library to support large counts for MPI communication functions
(https://github.com/jeffhammond/BigMPI).

In the course of actually trying to implement this the way the Forum
contends it can be done - i.e. using derived-datatypes - I have found
some issues that undermine the Forum's contention that it is so easy
for users to do it that it doesn't need to be in the standard.

To illustrate some of the issues that I have found, let us consider
the large-count implementation of nonblocking reduce...

# Example Use Case

It is entirely reasonable to think that some quantum chemist will want
to reduce a contiguous buffer of more than 2^31 doubles corresponding
to the Fock matrix if they have a multithreaded code, since 16 GB is
not an unreasonable amount of memory per node.

# Issues

Unlike the blocking case, where it is reasonable to chop up the data
and performance multiple operations (e.g.
https://github.com/jeffhammond/BigMPI/blob/master/src/reductions_x.c,
assuming that untested code is correct), one must return a single
request to the application if one is to implement MPIX_I(all)reduce_x
with the same semantics as MPI_Iallreduce, as I aspire to do in
BigMPI.

Issue #1: chopping doesn't work for nonblocking.

To do the large-count reduction in one nonblocking MPI call, a derived
datatype is required.  However, unlike in RMA, reductions cannot use
built-in ops for user-defined datatypes, even if they are trivially
composed of a large-count of built-in datatypes.  See
https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338 and
https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/34 for elaborate
commentary on why this semantic mismatch is lame.

Issue #2: cannot use built-in reduce ops.

Once we rule out using built-in ops with our large-count datatypes, we
must reimplement all of the reduction operations required.  I find
this to be nontrivial.  I have not yet figured out how to get at the
underlying datatype info in a simple manner.  It appears that
MPI_Type_get_envelope exists for this purpose, but it's a huge pain to
have to call this function when all I need to know is the number of
built-in datatypes so that I can apply my clever and use
MPI_Reduce_local inside of my user-defined operation.

Issue #3: implementing the user-defined reduce op isn't easy (in my opinion).

Many MPI implementations optimize reductions.  On Blue Gene/Q, MPI has
explicitly vectorized intrinsic/assembly code.  Unless
MPI_Reduce_local hits that code path, I am losing a huge amount of
performance in reductions when I go from 2^31 to 2^31+1 elements.  I
would not be surprised at all if user-defined ops+datatypes exercises
suboptimal code paths in many MPI implementations, which means that
the performance of nonblocking reductions is unnecessarily crippled.

Issue #4: inability to use optimizations in the MPI implementation.

# Conclusion

I believe this problem is best addressed in one of two ways:

1) Approve the semantic changes requested in tickets 34 and 338 so
that one can use built-in ops with homogeneous user-defined datatypes.
 This is my preference for multiple reasons.

2) Add large-count reductions to the standard.  This means 8 new
functions: blocking and nonblocking (all)reduce and
reduce_scatter(_block).  We don't need large-count functions for any
other collectives because the datatype solution works just fine there,
as I've already demonstrated in BigMPI
(https://github.com/jeffhammond/BigMPI/blob/master/src/collectives_x.c).

# Social Commentary

>From now on, when the Forum punts on things and says it's no problem
for users to roll their own using the existing functionality in MPI,
we should strive to be a bit more diligent and actually prototype that
implementation in a manner that proves how easy it is for users.  It
turns out, writing code for some things is harder than just talking
about them in a conference room...

# Related

https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338#comment:9
captures some of this feedback in Trac.

I created https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/423 for
the reasons described therein.

--
Jeff Hammond
jeff.science at gmail.com
_______________________________________________
mpi-forum mailing list
mpi-forum at lists.mpi-forum.org
http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum