<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">HI Dan,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Oh I like this decision tree – responses interleaved below<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Dan Holmes <danholmes@chi.scot><br>
<b>Date: </b>Friday, February 19, 2021 at 12:16 PM<br>
<b>To: </b>MPI Sessions working group <mpiwg-sessions@lists.mpi-forum.org><br>
<b>Cc: </b>"Pritchard Jr., Howard" <howardp@lanl.gov><br>
<b>Subject: </b>[EXTERNAL] Re: [mpiwg-sessions] more excitement - more nuanced response to issue 435<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in">Hi Howard, <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">My initial impression (from reading your email but not looking at the PDF yet), is:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">* I much prefer Rolf’s suggested reference to generic/unspecified “collective operations” rather than nailing it down to MPI_Ialltoall.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">* I don’t like the restriction that the user must finalise sessions in a particular order to match the internal implementation of a single session finalise at some remote process (e.g. the scenario of Rolf’s case
A on issue 435).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">More fundamental: we need a decision tree to tease apart the design decisions we are making at pace and with no reference implementation.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">First choice: does MPI_SESSION_FINALISE do anything non-local? If so, what?<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#00B0F0">If we think mpi_session_finalize should clean up what the user didn’t do, then for most interconnects we’d need some mechanism for insuring pending sends, etc. got off on the wire. Otherwise I can’t think of
anything from an implementation point of view.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#00B0F0"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">If no, then next choice is: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Root Q1: Do we wish to mandate that the user must do clean up prior to MPI_SESSION_FINALISE? If so, then eek! Breach of rule of least astonishment.<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#00B0F0">No but cleaning up should be recommended best practice<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Branch Q2: If no, then do we wish to mandate that MPI_SESSION_FINALISE does whatever clean up has not been done by the user? If so, eek! Significant change to accepted text.<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#00B0F0">We do have this sentence starting out the function description currently - This routine cleans up all MPI state associated with the supplied session. Up for interpretation what that means I guess. This alone
doesn’t imply anything about non-local.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#00B0F0"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">Branch Q3: If no, then does MPI_SESSION_FINALISE do anything non-local? If so, eek! What does it do? Panic.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#00B0F0">If no then there ought not to be anything non-lcocal<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">Branch Q4: If no, then does MPI_SESSION_FINALISE need to be defined as collective? If so, eek! Why? Why does it need that semantic? Panic.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#00B0F0">No in this event<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Branch Q5: If no, <span style="color:black">then does MPI_SESSION_FINALISE need to be defined as non-local? If so, eek! Why? Why does it need that semantic? Panic.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#00B0F0">Again no if we take Q4 answer to be negative.<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Branch Q6: If no, <span style="color:black">then we should define MPI_SESSION_FINALISE as local (meaning weak-local, of course)? If so, strike all text about collective operation(s) of any kind and strike any restriction
on ordering of calls and strike any restriction on the permitted associations/derivations of communicators from sessions.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#00B0F0">agreed<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><br>
<br>
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">This is a linear decision tree that leads to:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><br>
<br>
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">"</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">MPI_SESSION_FINALIZE is a local procedure; it does not free MPI objects derived from the session. It is erroneous to use MPI objects derived from a session after calling MPI_SESSION_FINALIZE
for that session.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><br>
<br>
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">If the user wishes to recover resources from MPI objects derived from a session, then appropriate calls to MPI procedures must be made by the user prior to calling MPI_SESSION_FINALIZE,
such as MPI_COMM_DISCONNECT (from communicators), MPI_WIN_FREE (for windows), and MPI_FILE_CLOSE (for files).</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">“</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><br>
<br>
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">Discuss.</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Helvetica;color:black"><br>
Cheers,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Helvetica;color:black">Dan.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Helvetica;color:black">—<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Helvetica;color:black">Dr Daniel Holmes PhD<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">Executive Director<br>
Chief Technology Officer<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Helvetica;color:black">CHI Ltd<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Helvetica;color:black"><a href="mailto:danholmes@chi.scot">danholmes@chi.scot</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Helvetica;color:black"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-left:.5in">On 19 Feb 2021, at 18:13, Pritchard Jr., Howard via mpiwg-sessions <<a href="mailto:mpiwg-sessions@lists.mpi-forum.org">mpiwg-sessions@lists.mpi-forum.org</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in">HI All,<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Ah this is exciting. So I spent some time on baking verbiage to add about MPI_Session_finalize<span class="apple-converted-space"> </span><i>non-local</i><span class="apple-converted-space"> </span>behavior.<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">See the attached cutout from the results pdf.<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I’ve added verbiage describing the semantics (copying some wording from MPI_Sendrecv, or at least the flavor) of session finalize in the event that the user has not cleaned up MPI objects associated with the session(s).<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">It’s a simple easy to understand (I think) model. Basically session finalize has the semantics of a MPI_Ialltoall for each communicators still associated with the session at finalize, followed by a waitall. As
long as all other processes finalizing their sessions generate in aggregate, a message pattern which matches, no deadlock. If not, potential deadlock. One takeaway from this is that we can’t support arbitrary associations of communicators to sessions in
each MPI process when the app doesn’t do its own cleanup so as to make MPI_Session_finalize a local op.<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I’ve added some examples and we can add more as we think needed. May have to change the presentation mechanism however.<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">I didn’t want to open this as a PR at this point, hence this notification mechanism.<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">Howard<span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in">-- <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"> <span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt">Howard Pritchard</span><span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt">HPC-ENV</span><span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.5pt">Los Alamos National Laboratory</span><span style="font-size:12.0pt"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:12.0pt"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="margin-left:.5in"><temp.pdf><span style="font-size:9.0pt;font-family:Helvetica">_______________________________________________<br>
mpiwg-sessions mailing list<br>
</span><a href="mailto:mpiwg-sessions@lists.mpi-forum.org"><span style="font-size:9.0pt;font-family:Helvetica">mpiwg-sessions@lists.mpi-forum.org</span></a><span style="font-size:9.0pt;font-family:Helvetica"><br>
</span><a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions"><span style="font-size:9.0pt;font-family:Helvetica">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
</div>
</body>
</html>