[MPI3 Fortran] One more MPI Fortran thing...

Mon Jan 25 11:22:02 CST 2010

Puri -- are you involved with the MPI-3 hybrid working group?  I have not been following it closely -- are they already talking about such issues?

On Jan 22, 2010, at 2:47 PM, Bill Long wrote:

> 
> 
> Craig Rasmussen wrote:
> > This is a good question that Bill Long should respond to.
> >
> > Bill has long claimed that Cray's coarray implementation works fine with their MPI library.  I'm not sure what this means precisely. 
> Certainly there is no problem with using the MPI API with a coarray
> program that I can think of because MPI would just do send/recv using
> local memory
> (but I can imagine some interesting race conditions if users aren't
> careful).
> I would guess that mpirun would have to know it is running a coarray
> program so that the two runtimes don't interfere with each other.
> >
> 
> It is not necessarily the case that there would be "two runtimes".
> Vendors have done the work for MPI  to launch and execute a parallel
> program consisting of replicated instances of the program running
> concurrently and communicating with each other.  Fortran 2008 coarrays
> use the same basic model.  There is some additional infrastructure
> needed for Fortran (and UPC), but this is incremental, not conflicting.
>   It is quite reasonable, though not required by the Fortran standard,
> that the vendor would just enhance their current runtime and use it for
> all of MPI, Fortran 2008, and UPC.  Cray calls the program launcher
> 'aprun' rather than 'mpirun', promoting the idea that the same
> infrastructure is used for all.  Such reuse saves development costs,
> which the bean counters like, and has the desirable side effect of
> facilitating execution of mixed codes that include any combination of
> MPI, Fortran 2008, or UPC (or SHMEM, for that matter).   I don't think
> any of the relevant standards specify this sort of commonality, but I
> expect customers to effectively require this capability.
> 
> 
> > I think running a hybrid MPI and coarray program will be common as MPI has more functionality at the moment (and probably always as it is library based).
> >
> 
> I agree, but I think 'hybrid' needs some clarification.  I see two,
> rather distinct cases:
> 
> 1) Multi-level parallelism.   Today this is characterized by MPI
> programs that use OpenMP within each rank to implement an additional
> layer of local, shared-memory parallelism.   This has been around for
> years.   It is quite reasonable to expect the use of OpenMP within a
> Fortran 2008 image in just the same way.  Several years ago I saw a
> comparison of running N MPI ranks with M-way OpenMP parallelism within
> each rank  and the same code with OpenMP turned off and run only with
> MPI using N*M ranks (same total number of processors).  In the past, the
> all-MPI version tended to have better performance.  However, with the
> more recent trends in core counts within a socket (4, 6, 8, 12, ...
> hundreds for a GPU) the balance is turning the other way.  I would
> expect to see more local, shared-memory parallelism within an MPI
> rank/Fortran image.  It might be from OpenMP (well established, mostly
> portable), automatic detection by the compiler (my preference), the new
> do concurrent construct in Fortran 2008 (also fine), or even CUDA/OpenCL
> (which I don't consider reasonable options for 'normal' programmers, but
> OK for BLAS-type library writers).
> 
> 2) Co-existing parallelism at the same level.  This would entail using
> Fortran 2008 parallelism with coarrays, or UPC, as an alternate
> mechanism to implement parallelism at the same level as MPI.  There is a
> one-to-one correspondence between the MPI rank and Fortran image / UPC
> process.  This could be going on without the programmer's knowledge (for
> example, parts of the Scalapack library might be written using
> coarrays), or by the programmer explicitly mixing the methods in the
> same program.
> 
> Assuming that Craig's comment was really about option (2), I would
> expect that this will be very common.  MPI has been around for many
> years, it has proven to be the most successful scheme so far for
> parallelism that does not reply on shared memory, and hundreds
> (thousands?) of programmer-centuries have been invested in creating huge
> MPI-based codes.  With its combination of success and inertia, MPI is
> not going away for a very very long time, if ever. And it is not likely
> that a large MPI code will suddenly be scrapped and rewritten entirely
> with Fortran 2008.   Instead, initially some section of a code might be
> rewritten using coarrays, leaving the rest of the code alone.  This has
> already happened for users with early access to coarrays.   This is a
> quite reasonable scheme for gradually evolving a code.  (An potential
> candidate that comes to mind is in WRF,  where the transfer of a very
> large data structure is coded as hundreds of calls to 'pack' routines
> followed by a bulk MPI Send call for the resulting buffer, and on the
> receive side a mirror set of 'unpack' routines.  The coarray alternative
> is a single assignment statement.)
> 
> As suggested by the end of Craig's comment, the reverse evolution is
> also possible.  A new code might be developed using coarrays, but might
> call a highly optimized MPI routine for a particular collective
> communication pattern.    This could be explicit, or hidden in a set of
> wrapper library routines that internally take advantage of the work that
> has gone into MPI optimization.    The Fortran committee has discussed
> the inclusion of collective intrinsics in the standard (the topic is
> currently deferred to a TR).  One school of though is that at least the
> bulk of these should be provided by a LaPack-style library outside the
> language spec.   If we go that route (currently a minority view), a
> vendor might very well leverage the existing MPI work inside the library.
> 
> 
> Cheers,
> Bill
> 
> 
> 
> > -craig
> >
> > On Jan 21, 2010, at 6:42 PM, Purushotham Bangalore wrote:
> >
> >> The other issue that I remember after the meeting: Since co-arrays will be part of the F08 standard, are they any potential issues that we need to be aware of? Or will the solutions from the Hybrid programming WG solve any potential problems that might arise?
> >>
> >> Thanks
> >> Puri
> >> ----- Original Message -----
> >> From: "Jeff Squyres" <jsquyres at cisco.com>
> >> To: "Craig Rasmussen" <crasmussen at newmexicoconsortium.org>
> >> Cc: "Craig Rasmussen" <crasmussen at lanl.gov>, "Purushotham Bangalore" <puri at cis.uab.edu>
> >> Sent: Thursday, January 21, 2010 7:04:05 PM
> >> Subject: One more MPI Fortran thing...
> >>
> >> Craig -- I forgot to mention this on the phone today: Puri asked in the Atlanta meeting today if we need to say anything about F08 in MPI-3.  I.e., do we need a paragraph/page/section/whatever about any issues that we expect might occur with using the explicit interface with F08 compilers?
> >>
> >> --
> >> Jeff Squyres
> >> jsquyres at cisco.com
> >>
> >
> >
> > _______________________________________________
> > mpi3-fortran mailing list
> > mpi3-fortran at lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran
> 
> --
> Bill Long                                           longb at cray.com
> Fortran Technical Support    &                 voice: 651-605-9024
> Bioinformatics Software Development            fax:   651-605-9142
> Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101
> 
> 
> _______________________________________________
> mpi3-fortran mailing list
> mpi3-fortran at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran
> 

-- 
Jeff Squyres
jsquyres at cisco.com