[mpiwg-rma] FENCE local completion requirements

[Balaji, Pavan]

On Sep 29, 2014, at 11:03 PM, Rajeev Thakur <thakur at mcs.anl.gov> wrote:

>> Let me rephrase —
>> 
>> Is the sentence in the standard to be read as: if there are preceding PUT/GET operations, it *will* be a barrier?
>> 
>> I don’t think the sentence says that.  I agree that an MPI implementation has to do that, but the standard is not explicit about it.
>> 
>> — Pavan
> 
> Yes, that's what the standard says. And it says that because the implementation has no other choice if it has to maintain fence semantics.
> 
> The standard says
> 
> "A fence call usually entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call."
> 
> Then it qualifies the "usually" by giving the case where it doesn't act as a barrier:
> 
> "However, a call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier."
> 
> Which means in all other cases, it acts as a barrier.

The right way to phrase that would have been:

A call to MPI_WIN_FENCE that ends an epoch entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call.  A call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier.

If this was the intention, it should be fixed in the standard.

Now to my next point — barrier synchronization only means that other processes have entered WIN_FENCE, not returned from WIN_FENCE.  This means that they might not have completed all operations at the target.

If we change my initial example to the following:

MPI_WIN_FENCE(win1)
MPI_PUT(win1)
MPI_WIN_FENCE(win1)

MPI_WIN_LOCK(win2)
MPI_GET(win2)
MPI_WIN_UNLOCK(win2)

.. the GET might not see the data written by PUT, since the WIN_FENCE only guarantees local completion and barrier semantics of the second FENCE only ensures that the target has “called” FENCE, not “returned" from FENCE.

  — Pavan

--
Pavan Balaji  ✉️
http://www.mcs.anl.gov/~balaji