<div dir="ltr">I tend to agree with this, the wording is misleading, but not incorrect.<div><br></div><div>There is a third implementation that has been only implicitly mentioned. The synchronization that the spec requires to start the access epoch is weaker than a barrier. A point-to-point synchronization between origin and target is what is required by the standard. This is achieved with the deferred implementation or potentially by piggybacking when RMA is done through eager send/recv (which is not such a bad option for active target). If you want to use RDMA, you could initiate the access epoch with a point-to-point synchronization first (send/recv ping-pong, RDMA read of a flag on the window, etc), before accessing the target window.<div>
<br></div><div> ~Jim.</div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Feb 26, 2014 at 12:39 AM, Balaji, Pavan <span dir="ltr"><<a href="mailto:balaji@anl.gov" target="_blank">balaji@anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Assuming your interpretation is right, the statement is not incorrect. At this point it is just a matter of taste and not something the standard needs to fix anyway.<br>
<span class="HOEnZb"><font color="#888888"><br>
— Pavan<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Feb 25, 2014, at 11:36 PM, Rajeev Thakur <<a href="mailto:thakur@mcs.anl.gov">thakur@mcs.anl.gov</a>> wrote:<br>
<br>
> I don't find it confusing. It makes the case where the barrier is not needed even more clear.<br>
><br>
> In the example below, if the second fence didn't have a NOPRECEDE, even a deferred implementation will need to do a barrier because it doesn't know if the process is a target of RMA operations from other processes.<br>
><br>
> If NOPRECEDE is called, it must be called on all processes, so then it is clear that no one is doing RMA.<br>
><br>
> So "in particular" might in fact be better than "for example".<br>
><br>
><br>
> On Feb 25, 2014, at 11:25 PM, "Balaji, Pavan" <<a href="mailto:balaji@anl.gov">balaji@anl.gov</a>><br>
> wrote:<br>
><br>
>><br>
>> Why do you want that MPI_MODE_NOPRECEDE? What’s its purpose?<br>
>><br>
>> — Pavan<br>
>><br>
>> On Feb 25, 2014, at 11:24 PM, Rajeev Thakur <<a href="mailto:thakur@mcs.anl.gov">thakur@mcs.anl.gov</a>> wrote:<br>
>><br>
>>> Instead of "in particular" if it said "for example, a call with assert equal to MPI_MODE_NOPRECEDE" would it help?<br>
>>><br>
>>> On Feb 25, 2014, at 11:14 PM, "Balaji, Pavan" <<a href="mailto:balaji@anl.gov">balaji@anl.gov</a>><br>
>>> wrote:<br>
>>><br>
>>>><br>
>>>> If that was indeed the intention, then the wording is very unfortunate. I’d recommend not having the mention of MPI_MODE_NOPRECEDE at all. It serves no real purpose and ends up confusing things.<br>
>>>><br>
>>>> — Pavan<br>
>>>><br>
>>>> On Feb 25, 2014, at 11:09 PM, Rajeev Thakur <<a href="mailto:thakur@mcs.anl.gov">thakur@mcs.anl.gov</a>> wrote:<br>
>>>><br>
>>>>> It means to say that in the example below, the second fence (which does not end any epoch) does not necessarily act as a barrier. The NOPRECEDE makes it very clear (and hence the "in particular a call with assert=NOPRECEDE") that no RMA calls were issued prior to that fence.<br>
>>>>><br>
>>>>> Fence<br>
>>>>> x=1<br>
>>>>> Fence(NOPRECEDE)<br>
>>>>> Put<br>
>>>>> Fence<br>
>>>>><br>
>>>>> Note that a fence "completes an RMA access epoch if it was preceded by another fence call and the local process issued RMA communication calls on win between these two calls. The call completes an RMA exposure epoch if it was preceded by another fence call and the local window was the target of RMA accesses between these two calls."<br>
>>>>><br>
>>>>> So the second fence does not complete any epoch, which the user has made even more clear with the assert, and so it need not be a barrier.<br>
>>>>><br>
>>>>><br>
>>>>> On Feb 25, 2014, at 10:58 PM, "Balaji, Pavan" <<a href="mailto:balaji@anl.gov">balaji@anl.gov</a>><br>
>>>>> wrote:<br>
>>>>><br>
>>>>>><br>
>>>>>> Then the following sentence in the standard is weird:<br>
>>>>>><br>
>>>>>> "a call to MPI_WIN_FENCE that is known not to end any epoch (in particular a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier”<br>
>>>>>><br>
>>>>>> Why this special treatment for NOPRECEDE as no WIN_FENCE has barrier semantics?<br>
>>>>>><br>
>>>>>> — Pavan<br>
>>>>>><br>
>>>>>> On Feb 25, 2014, at 10:54 PM, Rajeev Thakur <<a href="mailto:thakur@mcs.anl.gov">thakur@mcs.anl.gov</a>> wrote:<br>
>>>>>><br>
>>>>>>> Agreed that the handshake can't be done in the RMA operation.<br>
>>>>>>><br>
>>>>>>> If you choose to implement fence eagerly, i.e., perform RMA ops as they are called, the first fence will have to be a barrier.<br>
>>>>>>><br>
>>>>>>> But if you choose to implement the deferred fence, the first fence and all RMA ops can be deferred until the fence that completes the epoch.<br>
>>>>>>><br>
>>>>>>><br>
>>>>>>> On Feb 25, 2014, at 10:46 PM, "Zhao, Xin" <<a href="mailto:xinzhao3@illinois.edu">xinzhao3@illinois.edu</a>><br>
>>>>>>> wrote:<br>
>>>>>>><br>
>>>>>>>> I agree with this. If the handshake is done in the first RMA operation after the fence, then that operation becomes blocking. In Standard P418 it says all communication calls are non-blocking.<br>
>>>>>>>><br>
>>>>>>>> Xin<br>
>>>>>>>> ________________________________________<br>
>>>>>>>> From: mpiwg-rma [<a href="mailto:mpiwg-rma-bounces@lists.mpi-forum.org">mpiwg-rma-bounces@lists.mpi-forum.org</a>] on behalf of Balaji, Pavan [<a href="mailto:balaji@anl.gov">balaji@anl.gov</a>]<br>
>>>>>>>> Sent: Tuesday, February 25, 2014 9:38 PM<br>
>>>>>>>> To: MPI WG Remote Memory Access working group<br>
>>>>>>>> Subject: Re: [mpiwg-rma] Inconsistency of MPI_WIN_FENCE semantic<br>
>>>>>>>><br>
>>>>>>>> In practice, it’ll need to have barrier semantics. Otherwise, PUT will need to be a two-sided operation to ensure that it’s not issued before the other process calls MPI_WIN_FENCE.<br>
>>>>>>>><br>
>>>>>>>> — Pavan<br>
>>>>>>>><br>
>>>>>>>> On Feb 25, 2014, at 9:21 PM, Rajeev Thakur <<a href="mailto:thakur@mcs.anl.gov">thakur@mcs.anl.gov</a>> wrote:<br>
>>>>>>>><br>
>>>>>>>>>> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier.<br>
>>>>>>>>><br>
>>>>>>>>> It only says "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". NOT "This requires MPI_WIN_FENCE that starts an epoch to act as an barrier."<br>
>>>>>>>>><br>
>>>>>>>>> Why does the fence have to act as a barrier. The handshake could be done when first RMA operation is called after the fence.<br>
>>>>>>>>><br>
>>>>>>>>> Rajeev<br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> On Feb 25, 2014, at 8:41 PM, "Zhao, Xin" <<a href="mailto:xinzhao3@illinois.edu">xinzhao3@illinois.edu</a>> wrote:<br>
>>>>>>>>><br>
>>>>>>>>>> Hi all,<br>
>>>>>>>>>><br>
>>>>>>>>>> There is an inconsistency of MPI_WIN_FENCE semantic in MPI 3.0 Standard that makes me confused:<br>
>>>>>>>>>><br>
>>>>>>>>>> (1) On P440-P441 it say that "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process". This requires MPI_WIN_FENCE that starts an epoch to act as an barrier.<br>
>>>>>>>>>><br>
>>>>>>>>>> (2) However, (1) contradict with the word at end of P441: "a call to MPI_WIN_FENCE that is known not to end any epoch (in particular a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier".<br>
>>>>>>>>>><br>
>>>>>>>>>> Should the word of (1) add: "when MPI_MODE_NOPRECEDE is not given"?<br>
>>>>>>>>>><br>
>>>>>>>>>> Thanks,<br>
>>>>>>>>>><br>
>>>>>>>>>> Xin<br>
>>>>>>>>>><br>
>>>>>>>>>> _______________________________________________<br>
>>>>>>>>>> mpiwg-rma mailing list<br>
>>>>>>>>>> <a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
>>>>>>>>>> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
>>>>>>>>><br>
>>>>>>>>> _______________________________________________<br>
>>>>>>>>> mpiwg-rma mailing list<br>
>>>>>>>>> <a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
>>>>>>>>> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
>>>>>>>><br>
>>>>>>>> _______________________________________________<br>
>>>>>>>> mpiwg-rma mailing list<br>
>>>>>>>> <a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
>>>>>>>> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
>>>>>>>> _______________________________________________<br>
>>>>>>>> mpiwg-rma mailing list<br>
>>>>>>>> <a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
>>>>>>>> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
>>>>>>><br>
>>>>>>> _______________________________________________<br>
>>>>>>> mpiwg-rma mailing list<br>
>>>>>>> <a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
>>>>>>> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
>>>>>><br>
>>>>>> _______________________________________________<br>
>>>>>> mpiwg-rma mailing list<br>
>>>>>> <a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
>>>>>> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> mpiwg-rma mailing list<br>
>>>>> <a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
>>>>> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> mpiwg-rma mailing list<br>
>>>> <a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
>>>> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
>>><br>
>>> _______________________________________________<br>
>>> mpiwg-rma mailing list<br>
>>> <a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
>>> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
>><br>
>> _______________________________________________<br>
>> mpiwg-rma mailing list<br>
>> <a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
>> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
><br>
> _______________________________________________<br>
> mpiwg-rma mailing list<br>
> <a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
<br>
_______________________________________________<br>
mpiwg-rma mailing list<br>
<a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
<a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br>
</div></div></blockquote></div><br></div>