[Mpi3-rma] RMA proposal 1 update

[Underwood, Keith D]

Not really - as long as the ack is sent after the message reaches an ordering point in the NIC, then hashing by address should be sufficient.  Even if the ack is sent when the DMA is complete, it doesn't fix the multi-rail issue, since PCIExpress is a posted-write model.

Keith

> -----Original Message-----
> From: Darius Buntinas [mailto:buntinas at mcs.anl.gov]
> Sent: Tuesday, May 18, 2010 9:27 AM
> To: MPI 3.0 Remote Memory Access working group
> Cc: Underwood, Keith D
> Subject: Re: [Mpi3-rma] RMA proposal 1 update
> 
> 
> That depends on if the NIC sends the ACK, when it receives the packet,
> when it adds the write on the DMA queue, or when the DMA is complete.
> Does anyone have the IB spec handy?
> 
> -d
> 
> On 05/17/2010 10:17 PM, Underwood, Keith D wrote:
> >> I had mentioned that IB has remote completion, but after the Forum
> >> Sayantan reminded me that IB's remote completion semantics are
> weaker
> >> than what MPI RMA would require. By remote completion, IB only talks
> >> about completion from the remote network adapters perspective, not
> the
> >> remote memory. So, in a case where there are multiple network
> adapters,
> >> the only way to know of remote completion is through software active
> >> messages, which has more overhead than the hardware giving a
> >> notification.
> >
> > Trimming the thread since this is orthogonal ;-)
> >
> > So, if the NIC thinks it is complete, how can it not be complete?  If
> you stuff something over the PCIExpress link, it is then ordered
> relative to anything coming behind it, so...  You are thinking about
> one NIC signaling remote completion, the source getting that remote
> completion, and then an access to the same memory location happening
> over the other NIC?  You don't need active messages to make that work.
> You can simply do your access striping over the rails based on a hash
> of the address.  That way, complete the NIC is truly complete.  If you
> truly want to be bullet proof, everybody may have to use the same
> hash...
> >
> > Keith
> >
> > _______________________________________________
> > mpi3-rma mailing list
> > mpi3-rma at lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma