[Mpi3-rma] RMA proposal 1 update

[Underwood, Keith D]

> >>> OK, as long as everything goes through the NIC.  Are we considering
> >>> direct load/stores differently from remote puts/gets in terms of
> >> completion?
> >>
> >> That's a good point. Even if you don't do direct load/stores, MPI
> >> implementations will want to optimize MPI_Puts/gets to local or
> shared
> >> memory windows on cache-coherent architectures by using load/stores
> >> internally.
> >
> > Which should still be ok, if you do it right.  By the time that a
> target can "know" that a source thinks all of the Puts are complete,
> you should have made it globally visible (PCIExpress is an ordered
> channel, unless you use the unordered option, which you wouldn't if you
> wanted to do things like, oh, tell the app that a message had been
> received and really mean it ;-)  Worst case you have to do something
> like send a message down each pipe as part of the barrier (or
> comparable event) that informs the target that the source has done a
> flush...
> 
> Let me add one more level of complication then :-). Two network
> adapters
> + shared memory. Let's say the Put went over NIC1 and a mutex went over
> NIC2 (get-accumulate). Then a local process checks the mutex and wants
> to access the data written by the put.
> 
> Theoretically, there's no ordering here. Note that the two NICs can be
> on two different PCIe slots and cause corruption.

Now you are just trying to be difficult... First, your scenario is not legal.  You have to call a local MPI_Lock()/MPI_Unlock() before that data is visible in the private window to allow loads and stores.  Even accessing that item that was Put over NIC1 is undefined until the source has done a completion operation.  

Even then, I think you are discussing an ordering problem that exists in the base standard:  completing an MPI_Unlock() implies remote completing.  Real remote completion.  Until MPI_Unlock() completes, there is no guarantee of ordering between anything.  MPI_flush() does not add to this issue.

Keith