[MPI3 Fortran] One more MPI Fortran thing...

Fri Jan 22 13:47:28 CST 2010

Craig Rasmussen wrote:
> This is a good question that Bill Long should respond to.
> 
> Bill has long claimed that Cray's coarray implementation works fine with their MPI library.  I'm not sure what this means precisely.  
Certainly there is no problem with using the MPI API with a coarray 
program that I can think of because MPI would just do send/recv using 
local memory
(but I can imagine some interesting race conditions if users aren't 
careful).
I would guess that mpirun would have to know it is running a coarray 
program so that the two runtimes don't interfere with each other.
> 

It is not necessarily the case that there would be "two runtimes". 
Vendors have done the work for MPI  to launch and execute a parallel 
program consisting of replicated instances of the program running 
concurrently and communicating with each other.  Fortran 2008 coarrays 
use the same basic model.  There is some additional infrastructure 
needed for Fortran (and UPC), but this is incremental, not conflicting. 
  It is quite reasonable, though not required by the Fortran standard, 
that the vendor would just enhance their current runtime and use it for 
all of MPI, Fortran 2008, and UPC.  Cray calls the program launcher 
'aprun' rather than 'mpirun', promoting the idea that the same 
infrastructure is used for all.  Such reuse saves development costs, 
which the bean counters like, and has the desirable side effect of 
facilitating execution of mixed codes that include any combination of 
MPI, Fortran 2008, or UPC (or SHMEM, for that matter).   I don't think 
any of the relevant standards specify this sort of commonality, but I 
expect customers to effectively require this capability.

> I think running a hybrid MPI and coarray program will be common as MPI has more functionality at the moment (and probably always as it is library based).
> 

I agree, but I think 'hybrid' needs some clarification.  I see two, 
rather distinct cases:

1) Multi-level parallelism.   Today this is characterized by MPI 
programs that use OpenMP within each rank to implement an additional 
layer of local, shared-memory parallelism.   This has been around for 
years.   It is quite reasonable to expect the use of OpenMP within a 
Fortran 2008 image in just the same way.  Several years ago I saw a 
comparison of running N MPI ranks with M-way OpenMP parallelism within 
each rank  and the same code with OpenMP turned off and run only with 
MPI using N*M ranks (same total number of processors).  In the past, the 
all-MPI version tended to have better performance.  However, with the 
more recent trends in core counts within a socket (4, 6, 8, 12, ... 
hundreds for a GPU) the balance is turning the other way.  I would 
expect to see more local, shared-memory parallelism within an MPI 
rank/Fortran image.  It might be from OpenMP (well established, mostly 
portable), automatic detection by the compiler (my preference), the new 
do concurrent construct in Fortran 2008 (also fine), or even CUDA/OpenCL 
(which I don't consider reasonable options for 'normal' programmers, but 
OK for BLAS-type library writers).

2) Co-existing parallelism at the same level.  This would entail using 
Fortran 2008 parallelism with coarrays, or UPC, as an alternate 
mechanism to implement parallelism at the same level as MPI.  There is a 
one-to-one correspondence between the MPI rank and Fortran image / UPC 
process.  This could be going on without the programmer's knowledge (for 
example, parts of the Scalapack library might be written using 
coarrays), or by the programmer explicitly mixing the methods in the 
same program.

Assuming that Craig's comment was really about option (2), I would 
expect that this will be very common.  MPI has been around for many 
years, it has proven to be the most successful scheme so far for 
parallelism that does not reply on shared memory, and hundreds 
(thousands?) of programmer-centuries have been invested in creating huge 
MPI-based codes.  With its combination of success and inertia, MPI is 
not going away for a very very long time, if ever. And it is not likely 
that a large MPI code will suddenly be scrapped and rewritten entirely 
with Fortran 2008.   Instead, initially some section of a code might be 
rewritten using coarrays, leaving the rest of the code alone.  This has 
already happened for users with early access to coarrays.   This is a 
quite reasonable scheme for gradually evolving a code.  (An potential 
candidate that comes to mind is in WRF,  where the transfer of a very 
large data structure is coded as hundreds of calls to 'pack' routines 
followed by a bulk MPI Send call for the resulting buffer, and on the 
receive side a mirror set of 'unpack' routines.  The coarray alternative 
is a single assignment statement.)

As suggested by the end of Craig's comment, the reverse evolution is 
also possible.  A new code might be developed using coarrays, but might 
call a highly optimized MPI routine for a particular collective 
communication pattern.    This could be explicit, or hidden in a set of 
wrapper library routines that internally take advantage of the work that 
has gone into MPI optimization.    The Fortran committee has discussed 
the inclusion of collective intrinsics in the standard (the topic is 
currently deferred to a TR).  One school of though is that at least the 
bulk of these should be provided by a LaPack-style library outside the 
language spec.   If we go that route (currently a minority view), a 
vendor might very well leverage the existing MPI work inside the library.

Cheers,
Bill

> -craig
> 
> On Jan 21, 2010, at 6:42 PM, Purushotham Bangalore wrote:
> 
>> The other issue that I remember after the meeting: Since co-arrays will be part of the F08 standard, are they any potential issues that we need to be aware of? Or will the solutions from the Hybrid programming WG solve any potential problems that might arise?
>>
>> Thanks
>> Puri
>> ----- Original Message -----
>> From: "Jeff Squyres" <jsquyres at cisco.com>
>> To: "Craig Rasmussen" <crasmussen at newmexicoconsortium.org>
>> Cc: "Craig Rasmussen" <crasmussen at lanl.gov>, "Purushotham Bangalore" <puri at cis.uab.edu>
>> Sent: Thursday, January 21, 2010 7:04:05 PM
>> Subject: One more MPI Fortran thing...
>>
>> Craig -- I forgot to mention this on the phone today: Puri asked in the Atlanta meeting today if we need to say anything about F08 in MPI-3.  I.e., do we need a paragraph/page/section/whatever about any issues that we expect might occur with using the explicit interface with F08 compilers?
>>
>> -- 
>> Jeff Squyres
>> jsquyres at cisco.com
>>
> 
> 
> _______________________________________________
> mpi3-fortran mailing list
> mpi3-fortran at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran

-- 
Bill Long                                           longb at cray.com
Fortran Technical Support    &                 voice: 651-605-9024
Bioinformatics Software Development            fax:   651-605-9142
Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101