<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:482045499;
mso-list-type:hybrid;
mso-list-template-ids:-564480204 134807567 134807577 134807579 134807567 134807577 134807579 134807567 134807577 134807579;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Jeff,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<ol style="margin-top:0cm" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span style="mso-fareast-language:EN-US">Yes<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span style="mso-fareast-language:EN-US">If only it were that simple<o:p></o:p></span></li></ol>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Your first quote is compromised by the example 11.17 that follows it: which order is the interleaved execution mimicking? MPI_SEND;MPI_RECV and MPI_RECV;MPI_SEND both result in deadlock (as stated
in that example). That sentence needs “interpretation” – your wording is slightly better, but it needs to be something like “as if the stages of the operations were executed atomically in some order”. The initiation and starting stages of both the send and
receive operations are local and can happen without dependence on anything else (in particular, without dependence on each other). Once that has happened, both operations are enabled and must complete in finite time. The execution outcome is “as if” each of
the blocking procedures were replaced with a nonblocking initiation and completion procedure pair and both initiation procedures were executed (in some order) before the completion procedures (in some order).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">The observation and clarification above is necessary, but not sufficient, for resolving the logically concurrent issue. It speaks to execution ordering, but not to message matching ordering.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">However, rather than mash everything together (we’ve been down that road, we know where it leads), we could consider the merits of just this adjustment on its own. We could call it “The two MPI thread-safety
rules conflict with each other.”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">Best wishes,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Dan.<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Jeff Hammond <jeff.science@gmail.com>
<br>
<b>Sent:</b> 10 May 2023 07:28<br>
<b>To:</b> MPI Forum <mpiwg-hybridpm@lists.mpi-forum.org><br>
<b>Cc:</b> Holmes, Daniel John <daniel.john.holmes@intel.com><br>
<b>Subject:</b> Re: [mpiwg-hybridpm] Meeting Tomorrow<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">"All MPI calls are thread-safe, i.e., two concurrently running threads may make MPI calls and the outcome will be as if the calls executed in some order, even if their execution is interleaved."<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">I’m going to continue to die on the hill that “as if executed in some order” constrains the implementation behavior to something equivalent to “MPI operations are initiated atomically” because otherwise one cannot be guaranteed that some
ordering exists. The text about logically concurrent merely explains the obvious to users: it is impossible to know in what order unsynchronized threads execute operations. The previous sentence makes it clear what is meant by logically concurrent, and it
is consistent with Chapter 11, i.e. it logically unordered:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">“...if the process is multithreaded, then the <u>semantics of thread execution may not define a relative order between two send operations executed by two distinct threads</u>. The operations are logically concurrent..."<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
I can’t provide the full history of the Intel instruction <a href="https://community.intel.com/legacyfs/online/drupal_files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf">ENQCMD</a> but it appears to address the problem of
a large number of semi-independent HW units initiating MPI operations in a manner compliant with the text above.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">As I have stated previously, it is possible to relax the constraint “as if executed in some order” with the addition of a new threading level, which Pavan proposed years ago as <a href="https://github.com/mpiwg-sessions/sessions-issues/wiki/2016-06-07-forum#notes-from-meeting-bold--specific-work-to-do">MPI_THREAD_CONCURRENT</a> (although
details are impossible to find at this point).<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Jeff<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">On 9. May 2023, at 17.41, Holmes, Daniel John via mpiwg-hybridpm <<a href="mailto:mpiwg-hybridpm@lists.mpi-forum.org">mpiwg-hybridpm@lists.mpi-forum.org</a>> wrote:<br>
<br>
Hi all,<br>
Unfortunately, I am double-booked for tomorrow’s HACC WG time slot – so my answer to the implied question below is “not yet”.<br>
The “logically concurrent isn’t” issue #117 is now accepted and merged into MPI-4.1 (take a moment to celebrate!) – but it just says “here be dragons”.<br>
Do we care enough to defeat those dragons?<br>
Argument FOR: as systems become more heterogeneous, an MPI process is likely to abstractly “contain” more semi-independent HW units that will want to communicate with other MPI processes, which will result in lots of logically concurrent MPI communication
operations – exactly the territory in which these dragons live.<br>
Argument AGAINST: we’ve been throwing brave warriors into this particular dragon fire for about a decade and we’ve only now convinced ourselves that the dragons do, in fact, exist. How many more volunteers do we have and do they have sufficiently pointy sticks?<br>
Best wishes,<br>
Dan.<br>
From: mpiwg-hybridpm <<a href="mailto:mpiwg-hybridpm-bounces@lists.mpi-forum.org">mpiwg-hybridpm-bounces@lists.mpi-forum.org</a>> On Behalf Of Jim Dinan via mpiwg-hybridpm<br>
Sent: 09 May 2023 15:15<br>
To: Hybrid working group mailing list <<a href="mailto:mpiwg-hybridpm@lists.mpi-forum.org">mpiwg-hybridpm@lists.mpi-forum.org</a>><br>
Cc: Jim Dinan <<a href="mailto:james.dinan@gmail.com">james.dinan@gmail.com</a>><br>
Subject: [mpiwg-hybridpm] Meeting Tomorrow<br>
Hi All,<br>
We had to reschedule the topic planned for tomorrow's meeting, so the agenda is now open. Please let me know if you have a topic you'd like to discuss. If we don't have a topic ahead of time, we will cancel.<br>
Thanks,<br>
~Jim.<br>
_______________________________________________<br>
mpiwg-hybridpm mailing list<br>
<a href="mailto:mpiwg-hybridpm@lists.mpi-forum.org">mpiwg-hybridpm@lists.mpi-forum.org</a><br>
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm</a><o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>