<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Jeff/Pavan,
<div class=""><br class="">
</div>
<div class="">How does MPI determine the difference between “in such cases” and “in other cases not covered by the preceding sentence”? When is MPI permitted to ignore the chronological order and when is it required to preserve it?</div>
<div class=""><br class="">
</div>
<div class="">Thus I think, in addition to this clarification, we should make it crystal clear what MPI is allowed/required to do in response to operations issued on different threads at different times.</div>
<div class=""><br class="">
</div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Option 1) make it explicitly/unambiguously clear by changing the text that both implementations are permitted and equally valid (mine & Jeff’s interpretation of the current text). However, this means that the user can *never* (portably) rely on MPI preserving
the chronological order, even when that order is enforced using thread synchronisation! This seems untenable/unacceptable to me. If the user cares about order, they must marshal all MPI calls onto a single thread! This tells me we must go for option 2.</div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
</div>
<div class=""><br class="">
</div>
<div class="">Option 2) require that MPI preserve the ordering that it sees in all circumstances, whatever that is and whatever the reason for that ordering is. If the user cares about ordering then the user can use thread synchronisation to force the order
that MPI sees, and then trust MPI to preserve that deterministic order all the way to the receiver. If the user does not care about order, then they omit the thread synchronisation and MPI will preserve the non-deterministic order it sees at the sender. This
also defines what happens if the MPI calls are actually interleaved. I believe this is the current implementation in MPICH (as referenced by Pavan) but may not apply to *all* MPI libraries. If there is an implementation that does not currently preserve the
chronological order (perhaps MVAPICH2), it will need to be changed to do so - possibly introducing additional overhead, such as sequence numbers, and reducing communication performance. The new INFO key mpi_assert_allow_overtaking gives MPI the permission
to remove this (additional) overhead.<br class="">
<div class=""><br class="webkit-block-placeholder">
</div>
<div class="">Suggestion:</div>
<div class=""><br class="">
</div>
<div class="">"The non-overtaking rule extends to send and receive operations issued by different threads in an MPI process. The chronological ordering of send operations, as seen by the MPI library, shall be preserved in the same manner as if the send operations
were issued in that sequence by a single thread in an MPI process. Similarly, the ordering of receive operations issued by different threads in an MPI process shall be preserved in the same manner as if they were issued by a single thread."</div>
<div class=""><br class="">
</div>
<div class="">Or something like that.</div>
<div class=""><br class="">
</div>
<div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Cheers,</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Dan.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Applications Consultant in HPC Research<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
—</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
—</div>
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 24 Nov 2018, at 17:45, Jeff Hammond <<a href="mailto:jeff.science@gmail.com" class="">jeff.science@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<div dir="auto" class="">Sure, that’s a good fix.</div>
</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Jeff</div>
<div class=""><br class="">
<div class="gmail_quote">
<div dir="ltr" class="">On Sat, Nov 24, 2018 at 9:40 AM Balaji, Pavan <<a href="mailto:balaji@anl.gov" class="">balaji@anl.gov</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto" class="">
<div class=""><br class="">
</div>
That sentence is taken out of context. It only makes sense when you place it in the right context. Something like this:
<div class=""><br class="">
</div>
<div class="">“<i style="background-color:rgba(255,255,255,0)" class="">the semantics of thread execution may not define a relative order between two send operations executed by two distinct threads. [In such cases] The operations are logically concurrent,
even if one physically precedes the other.”</i><br class="">
<br class="">
Would this alternate text work:</div>
<div class=""><br class="">
</div>
<div class=""><span style="background-color:rgba(255,255,255,0)" class="">“the semantics of threading runtime might or might not define a relative order between two send operations executed by two distinct threads<b class="">.</b> In such cases, unless the
user performs additional synchronization to explicitly order the operations, they are considered to be logically concurrent.”</span><br class="">
<br class="">
<div id="m_2856989369231531353AppleMailSignature" dir="ltr" class="">Sent from my iPhone</div>
</div>
</div>
<div dir="auto" class="">
<div class="">
<div dir="ltr" class=""><br class="">
On Nov 24, 2018, at 11:33 AM, Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank" class="">jeff.science@gmail.com</a>> wrote:<br class="">
<br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
On Nov 24, 2018, at 8:56 AM, Balaji, Pavan <<a href="mailto:balaji@anl.gov" target="_blank" class="">balaji@anl.gov</a>> wrote:<br class="">
<br class="">
</div>
<blockquote type="cite" class="">
<div class="">Jeff,
<div class=""><br class="">
</div>
<div class="">I’m OK with adding additional text to clarify it.</div>
<div class=""><br class="">
</div>
<div class="">FWIW, I still think the text is not ambiguous. In particular, it is simply warning the user that the thread execution may not define a relative order (as in, you might accidentally get some order because of the OS behavior). That does not mean
that one cannot achieve a relative order using additional synchronization outside of MPI.</div>
<div class=""><br class="">
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">How is the relative order outside of MPI on which you intend to rely not the physical order referenced in the following?</div>
<div class=""><br class="">
</div>
<div class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="gmail_quote"><i style="background-color:rgba(255,255,255,0)" class=""><font class="">The operations are logically concurrent, even if one physically precedes the other. </font></i></div>
</div>
</div>
</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""></div>
<div class="">In any case, let’s just add some text as you suggested below instead of arguing about it.</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">It’s hard to know what the right fix is when we cannot agree on whether there is a problem in the first place. </div>
<div class=""><br class="">
</div>
<div class="">Jeff</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class=""></div>
<div class=""> — Pavan<br class="">
<br class="">
<div id="m_2856989369231531353AppleMailSignature" dir="ltr" class="">Sent from my iPhone</div>
<div dir="ltr" class=""><br class="">
On Nov 24, 2018, at 10:38 AM, Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank" class="">jeff.science@gmail.com</a>> wrote:<br class="">
<br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div dir="ltr" class=""><br class="">
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="">On Fri, Nov 23, 2018 at 2:59 PM Balaji, Pavan <<a href="mailto:balaji@anl.gov" target="_blank" class="">balaji@anl.gov</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi Dan,<br class="">
<br class="">
> On Nov 23, 2018, at 4:11 AM, HOLMES Daniel <<a href="mailto:d.holmes@epcc.ed.ac.uk" target="_blank" class="">d.holmes@epcc.ed.ac.uk</a>> wrote:<br class="">
> However, it is *also* a correct implementation choice to ignore that “physical order” even in this case because the MPI library does not know, and cannot determine, *why* that “physical order” happened.<br class="">
<br class="">
I don't think this is a correct implementation and I'm not sure what part of the chapter is causing you to interpret this as a correct implementation. If there's algorithmic logic in the application to guarantee an order, then those operations are not logically
concurrent. Although I'm happy to help clarify something that's unclear in the standard, I'm at a loss as to what is unclear here.<br class="">
<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">As I included before, this is the relevant text:</div>
<div class=""><i style="font-family:CMR10;font-size:14.6667px" class=""><br class="m_2856989369231531353gmail-Apple-interchange-newline">
If a process has a single thread of execution, then any two communications executed by this process are ordered. On the other hand,
<b class="">if the process is multithreaded, then the semantics of thread execution may not define a relative order between two send operations executed by two distinct threads.</b> The operations are logically concurrent, even if one physically precedes the
other. In such a case, the two messages sent can be received in any order. Similarly, if two receive operations that are logically concurrent receive two successively sent messages, then the two messages can match the two receives in either order. </i><br class="">
</div>
<div class=""> </div>
<div class="">The problem with the text is that it does not state any means for the user to logically order operations on different threads. The explicit statement that physical order does not imply logical order means that users cannot rely on the order of
thread execution alone.</div>
<div class=""><br class="">
</div>
<div class="">The solution to this problem is to add text that indicates that the user can impart a logical order via thread synchronization primitives that order the execution of sends and weaken the problematic sentence such that it only applies when physical
ordering is coincidental and not the result of any synchronization between threads.</div>
<div class=""><br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
FWIW, every implementation of MPI that I know of interprets the standard the way I stated it, i.e., those operations are not concurrent and the MPI library has to process them in the order that it sees it. Whether that is an explicit scheduling done by the
user or is an accidental schedule created by the OS cannot be determined by the MPI library, so it better respect the order that it sees.<br class="">
</blockquote>
<div class=""><br class="">
</div>
<div class="">It would be good to look at MPI implementations that support multi-rail interconnects. How does MVAPICH2 mrail implement ordering in this case? Do they just use one rail per process or one rail per communicator?</div>
<div class=""><br class="">
</div>
<div class="">Jeff</div>
<div class=""> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
-- Pavan<br class="">
<br class="">
</blockquote>
</div>
<br clear="all" class="">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="m_2856989369231531353gmail_signature" data-smartmail="gmail_signature">
Jeff Hammond<br class="">
<a href="mailto:jeff.science@gmail.com" target="_blank" class="">jeff.science@gmail.com</a><br class="">
<a href="http://jeffhammond.github.io/" target="_blank" class="">http://jeffhammond.github.io/</a></div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
-- <br class="">
<div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br class="">
<a href="mailto:jeff.science@gmail.com" target="_blank" class="">jeff.science@gmail.com</a><br class="">
<a href="http://jeffhammond.github.io/" target="_blank" class="">http://jeffhammond.github.io/</a></div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>