<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><br></div><div><br>On Nov 24, 2018, at 8:56 AM, Balaji, Pavan <<a href="mailto:balaji@anl.gov">balaji@anl.gov</a>> wrote:<br><br></div><blockquote type="cite"><div>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
Jeff,
<div><br>
</div>
<div>I’m OK with adding additional text to clarify it.</div>
<div><br>
</div>
<div>FWIW, I still think the text is not ambiguous. In particular, it is simply warning the user that the thread execution may not define a relative order (as in, you might accidentally get some order because of the OS behavior). That does not mean that one
cannot achieve a relative order using additional synchronization outside of MPI.</div>
<div><br></div></div></blockquote><div><br></div><div>How is the relative order outside of MPI on which you intend to rely not the physical order referenced in the following?</div><div><br></div><div><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><i style="caret-color: rgb(0, 0, 0); background-color: rgba(255, 255, 255, 0);"><font color="#000000">The operations are logically concurrent, even if one physically precedes the other. </font></i></div></div></div></div><br><blockquote type="cite"><div><div>
</div>
<div>In any case, let’s just add some text as you suggested below instead of arguing about it.</div></div></blockquote><div><br></div><div>It’s hard to know what the right fix is when we cannot agree on whether there is a problem in the first place. </div><div><br></div><div>Jeff</div><br><blockquote type="cite"><div><div>
</div>
<div> — Pavan<br>
<br>
<div id="AppleMailSignature" dir="ltr">Sent from my iPhone</div>
<div dir="ltr"><br>
On Nov 24, 2018, at 10:38 AM, Jeff Hammond <<a href="mailto:jeff.science@gmail.com">jeff.science@gmail.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr"><br>
<br>
<div class="gmail_quote">
<div dir="ltr">On Fri, Nov 23, 2018 at 2:59 PM Balaji, Pavan <<a href="mailto:balaji@anl.gov">balaji@anl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi Dan,<br>
<br>
> On Nov 23, 2018, at 4:11 AM, HOLMES Daniel <<a href="mailto:d.holmes@epcc.ed.ac.uk" target="_blank">d.holmes@epcc.ed.ac.uk</a>> wrote:<br>
> However, it is *also* a correct implementation choice to ignore that “physical order” even in this case because the MPI library does not know, and cannot determine, *why* that “physical order” happened.<br>
<br>
I don't think this is a correct implementation and I'm not sure what part of the chapter is causing you to interpret this as a correct implementation. If there's algorithmic logic in the application to guarantee an order, then those operations are not logically
concurrent. Although I'm happy to help clarify something that's unclear in the standard, I'm at a loss as to what is unclear here.<br>
<br>
</blockquote>
<div><br>
</div>
<div>As I included before, this is the relevant text:</div>
<div><i style="font-family:CMR10;font-size:14.6667px"><br class="gmail-Apple-interchange-newline">
If a process has a single thread of execution, then any two communications executed by this process are ordered. On the other hand,
<b>if the process is multithreaded, then the semantics of thread execution may not define a relative order between two send operations executed by two distinct threads.</b> The operations are logically concurrent, even if one physically precedes the other.
In such a case, the two messages sent can be received in any order. Similarly, if two receive operations that are logically concurrent receive two successively sent messages, then the two messages can match the two receives in either order. </i><br>
</div>
<div> </div>
<div>The problem with the text is that it does not state any means for the user to logically order operations on different threads. The explicit statement that physical order does not imply logical order means that users cannot rely on the order of thread
execution alone.</div>
<div><br>
</div>
<div>The solution to this problem is to add text that indicates that the user can impart a logical order via thread synchronization primitives that order the execution of sends and weaken the problematic sentence such that it only applies when physical ordering
is coincidental and not the result of any synchronization between threads.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
FWIW, every implementation of MPI that I know of interprets the standard the way I stated it, i.e., those operations are not concurrent and the MPI library has to process them in the order that it sees it. Whether that is an explicit scheduling done by the
user or is an accidental schedule created by the OS cannot be determined by the MPI library, so it better respect the order that it sees.<br>
</blockquote>
<div><br>
</div>
<div>It would be good to look at MPI implementations that support multi-rail interconnects. How does MVAPICH2 mrail implement ordering in this case? Do they just use one rail per process or one rail per communicator?</div>
<div><br>
</div>
<div>Jeff</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
-- Pavan<br>
<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br>
<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br>
<a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div>
</div>
</blockquote>
</div>
</div></blockquote></body></html>