[Mpi3-rma] RMA proposal 1 update

[Jeff Hammond]

On Fri, May 21, 2010 at 3:27 PM, Underwood, Keith D
<keith.d.underwood at intel.com> wrote:
> I think (although I am not an ARMCI expert) that passive target with the lockall/unlockall and the flush/flushall extensions we have proposed gives you (approximately) ARMCI.  The question is then:  where does the allflushall to support GA belong?  ARMCI is actually an easier target than GA, because (like many models) ARMCI only has one-sided completion.  But, I thought you wanted better than ARMCI?

I completely agree with the first part.  All I'm asking is that, in
addition to flush and flushall, we add allflushall so that some of us
can defend against pathological behavior when GA_Sync calls flushall
from every node.  It is in this respect that I want better than ARMCI.
 ARMCI is effectively a p2p transport layer, where as GA's programming
model leads to much collective-like behavior.  It seems natural to
want MPI-3 to do a better job supporting GA than ARMCI.  Adding
allflushall is one step in this direction.

> The struggle here is that GA has both one-sided and collective completion.  Collective completion can obviously be emulated by one-sided completion + a barrier, but you have indicated that that is a performance issue.  Unfortunately, there is not an obvious place inside of the existing MPI RMA interface where the mixture of collective and one-sided completion fit.  The need for both one-sided and collective completion was clearly not architected into MPI one-sided and may very well break the architecture.  It certainly breaks the naming (there is nothing "passive" about a target that calls allfenceall and there is nothing "active" about a target when an initiator does one-side completion).

Clearly, allflushall is not passive, but it is a convenience operation
to complete what would otherwise be a passive-target epoch.  That it
would be inter-operable with all passive-target functionality, and
perhaps not start/post/complete (which still makes no sense to me),
makes it clearly part of the passive-target set.

> Does anybody know of another model (other than GA) that calls for a mixture of collective and one-sided completion?  CoArray Fortran uses collective completion, UPC expects one-sided completion, SHMEM only exposes one-sided completion, ARMCI only exposes one-sided completion...  If we could look at a second model that needed a mixture, it might help us formulate a better solution.

ARMCI has collective completion.  From
<http://www.emsl.pnl.gov/docs/parsoft/armci/documentation.htm#compl>:

int ARMCI_Barrier()
PURPOSE: Synchronize processors and memory. This operation combines
functionality of
        MPI_Barrier and ARMCI_AllFence.

I've been talking about allflushall because ARMCI_Barrier is partially
redundant with MPI_Barrier, which is clearly already in MPI-3.

Jeff

-- 
Jeff Hammond
Argonne Leadership Computing Facility
jhammond at mcs.anl.gov / (630) 252-5381
http://www.linkedin.com/in/jeffhammond