[mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory windows

Wed Feb 5 10:30:20 CST 2014

Jeff and all, 

it looks like that it works as MPI-3 is designed:

I need to add once at begin a MPI_WIN_LOCK_ALL(MPI_MODE_NOCHECK, win)
and once at end a MPI_WIN_UNLOCK_ALL(win)
and then all works fine with MPI_WIN_SYNC in each iteration.

Is this usage consistent with the definition in the MPI-3 standard?

Here the total scenario that I use:

--------------------
X is part of a shared memory window and should mean the same
memory location in both processes

Process A               Process B

MPI_WIN_LOCK_ALL(       MPI_WIN_LOCK_ALL(
MPI_MODE_NOCHECK,win)   MPI_MODE_NOCHECK,win) 

DO ...                  DO ...
  x=...
  MPI_F_SYNC_REG(X)
  MPI_Barrier             MPI_Barrier
                          MPI_F_SYNC_REG(X)
                          print X
END DO                  END DO

MPI_WIN_UNLOCK_ALL(win) MPI_WIN_UNLOCK_ALL(win)

--------------------

Best regards
Rolf

----- Original Message -----
> From: "Jeff Hammond" <jeff.science at gmail.com>
> To: "MPI WG Remote Memory Access working group" <mpiwg-rma at lists.mpi-forum.org>
> Cc: "Stefan Andersson" <stefan at cray.com>, "Bill Long" <longb at cray.com>
> Sent: Tuesday, February 4, 2014 7:42:58 PM
> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared memory	windows
> 
> "For the purposes of synchronizing the private and public window,
> MPI_WIN_SYNC has the effect of ending and reopening an access and
> exposure epoch on the window (note that it does not actually end an
> epoch or complete any pending MPI RMA operations)."
> 
> I think this is interpreted to mean that this call is only valid
> inside of an existing epoch and thus if you want to call it, you need
> to use it inside of a passive-target epoch.  Thus, it is not merely a
> portable abstraction for a memory barrier.
> 
> I think we should fix MPICH and/or MPI-Next to allow the more general
> use such that your code is standard-compliant and executes correctly.
> 
> I await violent disagreement from others :-)
> 
> Jeff
> 
> On Tue, Feb 4, 2014 at 12:34 PM, Rolf Rabenseifner
> <rabenseifner at hlrs.de> wrote:
> > Brian, Pavan, and Jeff,
> >
> > you convinced me. I did it, see attached file, and my mpich based
> > Cray lib tells
> >
> > Rank 0 [Tue Feb  4 19:31:28 2014] [c9-1c2s7n0] Fatal error in
> > MPI_Win_sync: Wrong synchronization of RMA calls , error stack:
> > MPI_Win_sync(113)...: MPI_Win_sync(win=0xa0000001) failed
> > MPIDI_Win_sync(2495): Wrong synchronization of RMA calls
> >
> > (only once in each process).
> >
> > I expect, that this is now an implementation bug that should be
> > fixed by mpich and cray?
> >
> > Best regards
> > Rolf
> >
> > ----- Original Message -----
> >> From: "Brian W Barrett" <bwbarre at sandia.gov>
> >> To: "MPI WG Remote Memory Access working group"
> >> <mpiwg-rma at lists.mpi-forum.org>
> >> Cc: "Stefan Andersson" <stefan at cray.com>, "Bill Long"
> >> <longb at cray.com>
> >> Sent: Tuesday, February 4, 2014 7:09:02 PM
> >> Subject: Re: [mpiwg-rma] [EXTERNAL] Re: Synchronization on shared
> >> memory windows
> >>
> >> On 2/4/14 11:01 AM, "Rolf Rabenseifner" <rabenseifner at hlrs.de>
> >> wrote:
> >>
> >> >The MPI_WIN_SYNC (not the Fortran MPI_F_SYNC_REG)
> >> >has no meaning in the unified memory model if all accesses
> >> >are done without RMA routines.
> >> >It has only a meaning if different public and privat copy is
> >> >there (MPI-3.0 p450:46-p451:2).
> >> >MPI-3.0 p456:3 - p457:7 define the rules for the unified memory
> >> >model
> >> >but there is no need to use MPI_WIN_SYNC.
> >>
> >> Right, there's no need from an MPI point of view, but that doesn't
> >> mean
> >> that the language/compiler/processor doesn't have a need for extra
> >> synchronization.
> >>
> >> >The combination of X=13 and MPI_F_SYNC_REG(X)
> >> >before MPI_Barrier should guarantee that all bytes of X are
> >> >stored in memory. The same should be valid in C,
> >> >because the C compiler has no chance to see whether
> >> >MPI_Barrier will access the bytes of X or not.
> >> >And if it is guaranteed to be in the unified memory,
> >> >then the other process (B) should be able to correctly
> >> >read the data after the return from its barrier.
> >> >
> >> >What is wrong with my thinking?
> >> >Which detail do I miss?
> >>
> >> According to my reading of the spec, MPI_F_SYNC_REG only prevents
> >> the
> >> language/compiler from moving the store, but does not say anything
> >> about
> >> processor ordering.  So the WIN_SYNC in my last e-mail will add
> >> the
> >> processor memory barrier, which will give you all the semantics
> >> you
> >> need.
> >>
> >> Shared memory programming is a disaster in most languages today,
> >> so
> >> we
> >> decided to pass that disaster on to the user.  We really can't
> >> help,
> >> without adding lots of overhead (ie, using put/get/rma
> >> synchronization).
> >> So if a user already knows how to do shared memory programming,
> >> this
> >> will
> >> feel natural.  If they don't, it's going to hurt badly :/.
> >>
> >>
> >> Brian
> >>
> >> --
> >>   Brian W. Barrett
> >>   Scalable System Software Group
> >>   Sandia National Laboratories
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> mpiwg-rma mailing list
> >> mpiwg-rma at lists.mpi-forum.org
> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma
> >>
> >
> > --
> > Dr. Rolf Rabenseifner . . . . . . . . . .. email
> > rabenseifner at hlrs.de
> > High Performance Computing Center (HLRS) . phone
> > ++49(0)711/685-65530
> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 /
> > 685-65832
> > Head of Dpmt Parallel Computing . . .
> > www.hlrs.de/people/rabenseifner
> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room
> > 1.307)
> >
> > _______________________________________________
> > mpiwg-rma mailing list
> > mpiwg-rma at lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma
> 
> 
> 
> --
> Jeff Hammond
> jeff.science at gmail.com
> _______________________________________________
> mpiwg-rma mailing list
> mpiwg-rma at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma
> 

-- 
Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)