[Mpi3-rma] RMA proposal 1 update

[Pavan Balaji]

On 05/18/2010 01:25 PM, Underwood, Keith D wrote:
>> On 05/18/2010 10:58 AM, Darius Buntinas wrote:
>>> OK, as long as everything goes through the NIC.  Are we considering
>>> direct load/stores differently from remote puts/gets in terms of
>> completion?
>>
>> That's a good point. Even if you don't do direct load/stores, MPI
>> implementations will want to optimize MPI_Puts/gets to local or shared
>> memory windows on cache-coherent architectures by using load/stores
>> internally.
> 
> Which should still be ok, if you do it right.  By the time that a target can "know" that a source thinks all of the Puts are complete, you should have made it globally visible (PCIExpress is an ordered channel, unless you use the unordered option, which you wouldn't if you wanted to do things like, oh, tell the app that a message had been received and really mean it ;-)  Worst case you have to do something like send a message down each pipe as part of the barrier (or comparable event) that informs the target that the source has done a flush... 

Let me add one more level of complication then :-). Two network adapters 
+ shared memory. Let's say the Put went over NIC1 and a mutex went over 
NIC2 (get-accumulate). Then a local process checks the mutex and wants 
to access the data written by the put.

Theoretically, there's no ordering here. Note that the two NICs can be 
on two different PCIe slots and cause corruption.

  -- Pavan

-- 
Pavan Balaji
http://www.mcs.anl.gov/~balaji