<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:251090193;
mso-list-template-ids:1952064668;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1
{mso-list-id:406149697;
mso-list-type:hybrid;
mso-list-template-ids:607786702 134807573 134807577 134807579 134807567 134807577 134807579 134807567 134807577 134807579;}
@list l1:level1
{mso-level-number-format:alpha-upper;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Ralph,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thanks for the question. I’m not sure I fully see the difficulty.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<ol start="1" type="A">
<li class="MsoListParagraph" style="mso-list:l1 level1 lfo2"><span style="mso-fareast-language:EN-US">Many failures in one “event” means that when we go around the while loop and try to rebuild, we discover a whole bunch of failures all at once. The MPI_Session_get_failed
procedure gives a big group with size >>1. We exclude all of those processes in one group difference call and we’re back to a simple case. I’m assuming the detectors are well-behaved and all of them detect the same event and notify the same failures. The timing
might not be perfect, so some detectors might give a big group, whereas others give a small group. That will lead to some processes discovering many failures during the operation – that’s the next case.<o:p></o:p></span></li><li class="MsoListParagraph" style="mso-list:l1 level1 lfo2"><span style="mso-fareast-language:EN-US">Many failures being detected during the operation definitely slows it down, but I don’t see any kind of undecidable or livelock/deadlock outcome. It looks
like O(P log(P)) is a sensible worst-case asymptotic, even when the failures are happening one at a time at just the wrong moments. One event taking out a bunch of processes at the same time seems to be an easier case than that. Eventually every node of the
spanning tree will be traversed, either directly or by proxy via another live process, so all the failures would be discovered by all live processes. The agree catches the case where live processes fail after they’ve finished collaborating with the other processes
(when they believe they have complete information but before they return to the user).<o:p></o:p></span></li></ol>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Every failed involved process will be discovered by a live involved process unless all the involved processes that believed the failed involved process was involved in the operation fail before or
during the operation; in which case, it no longer matters. The survivors eventually come to consensus on the membership of the survivors group by excluding any and all processes that the survivors cannot unanimously agree are alive and involved.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">The key rule here is that “if any process believes that procF has failed, then all other processes must accept that decision and treat procF as failed also” which is conservative but fits with the
process fail-stop model.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">The only serious problem I can foresee is bifurcation: two or more closed compact graphs of processes that are disjoint, separately coming to consensus that they are the only survivors because every
link between those disjoint groups has been severed by process failures. We discussed a similar situation (network bifurcation) at length in a previous meeting, but there was no consensus on the correct response from MPI. One school of thought permits all
the disjoint groups to carry on, oblivious of the others. Another school of thought dictates that every group should attempt to destroy all the others to ensure that it really is the only surviving group. Neither outcome is ideal – so let’s pretend bifurcation
does not happen! We can hope to get away with that unsavoury decision for the network bifurcation case because “network bifurcation” is not a process fail-stop fault and is, therefore, out of scope for ULFM. However, the bifurcation because of a set of strategically
placed process failures is definitely in scope and just as tricky.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Do you see additional problematic situations?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Best wishes,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Dan.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> mpiwg-sessions <mpiwg-sessions-bounces@lists.mpi-forum.org>
<b>On Behalf Of </b>Ralph Castain via mpiwg-sessions<br>
<b>Sent:</b> Thursday, February 1, 2024 7:13 PM<br>
<b>To:</b> MPI Sessions working group <mpiwg-sessions@lists.mpi-forum.org><br>
<b>Cc:</b> Ralph Castain <rhc@pmix.org><br>
<b>Subject:</b> Re: [mpiwg-sessions] Sessions WG - meet 1/29/24<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Just curious - how do you figure to handle the cascade-of-failure scenario? This is by far the most common as you either lose a node (which means the detection of individual failure for each proc on that node) or you lose a switch (and
therefore get report of individual failure for all procs communicating over that device). Kind of rare for a proc to just die on its own (except for contrived tests, of course), though not impossible.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Ralph<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Feb 1, 2024, at 10:02 AM, Holmes, Daniel John via mpiwg-sessions <<a href="mailto:mpiwg-sessions@lists.mpi-forum.org">mpiwg-sessions@lists.mpi-forum.org</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Hi all,<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">WORDS to capture our discussion on Monday 29th Jan.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">The basic idea in the example code I gave is to detect process fail-stop faults, shrink the active communicator to exclude the failed processes, and then carry on.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">The while loop captures the "carry on" part, only exit when we hit the break statement (or we suffer a process fail-stop fault).<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">The additional code block captures the "shrink" part by creating a new communicator that excludes failed processes.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">The MPI_Group_from_session_pset calls will all produce identical groups, even after processes have failed. This is a local procedure and there's no reason for it to return an error. We should guarantee that it never returns errors of class
MPI_ERR_PROC_FAILED.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">The MPI_Session_get_proc_failed calls give a snapshot of the knowledge contained in the local detector. There is no communication or nonlocal dependence here. This is a (new) local procedure and there's no reason for it to return an error.
We should guarantee that it never returns errors of class MPI_ERR_PROC_FAILED.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">The group manipulation procedures are existing MPI and will work exactly the same as they always have done. These are local procedures and there's no reason for them to return an error. We should guarantee that they never return errors
of class MPI_ERR_PROC_FAILED.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">The devils are always in the details. The details in this case are all in the implementation of MPI_Comm_create_from_group.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">The MPI_Comm_create_from_group procedure must handle some difficult cases:<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">1. the potential for failed processes to exist in the group that is passed in by the calling MPI process<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">2. the potential for the group passed in to be different to the groups passed in by other MPI processes<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">3. the potential for the additional MPI processes to fail during the operation<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">1. failed before the operation<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">If this MPI process attempts to communicate with a failed process, it's detector must eventually detect that, otherwise it is broken!<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">This means that the failure of the other MPI process is discovered during the operation -- see point (3).<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">2. different groups at different processes<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">There are several sub-cases here:<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">a) procA's group includes procB but procB has failed (any time before procA communicates with it)<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">b) procA's group contains procB but procB's does not include procA<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">c) procA's group contains procB and procC but procB's group includes procA but does not include procC<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">d) procA's group and procB's group are identical but one of them later discovers an (a|b|c) problem<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Current shrink implementations rely on shared knowledge of the survivors so each involved process can independently create the same spanning tree as all the other involved processes.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">This simplifies the design and results in a better worst-case performance scaling, at least asymptotically [ED: pls check]<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">All of the above sub-cases can happen during a shrink operation, so intuitively this is no worse than that.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">We have two broad categories of approach here:<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">- any problem results in an error, return immediately without creating the output communicator<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">- any FT problem is dealt with internally; some kind of best-effort communicator is created<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">We probably want a resilient algorithm that doesn’t have dreadful scaling.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">3. failures during the operation<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">We can just do an agreement at the end to catch failures that happen after the critical moment when other processes communicate with it.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Resilient algorithm:<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Start optimistic: Create a spanning tree from the group members you have been given in the call.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Be eager to help: Always listen for incoming protocol messages related to communicator creation.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Attempt communication with your direct children;<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> if new failure detected, update the local group, update the local detector, fix the spanning tree (skip that child, but add that child's children as your direct children)<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> if a child reports a different group (compare hashes), figure out the difference, update the local detector, fix the spanning tree (recreate it using only uncontacted undead processes)<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Attempt communication with your parent;<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> if new failure detected, update the local group, update the local detector, fix the spanning tree (skip the parent, but add that parent's parent as your direct parent)<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> if the parent reports a different group (compare hashes), figure out the difference, update the local detector, fix the spanning tree (recreate it using only uncontacted undead processes)<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Assume it all worked beautifully but execute MPI_Comm_agree to make certain.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> if new failure detected, update the local group, update the local detector, start again<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> if the agreement succeeds without discovering new failures, we're done!<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Best-case asymptotic scaling is O(log(p)), a spanning tree traversal of P processes.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Worst-case O(sum_i=1,p-1(log(p-i)) < O(p log(p)), no more than p-1 tree traversals with a shrinking tree size.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Best wishes,<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Dan.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-color:currentcolor currentcolor;border-image: none">
<div>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span class="apple-converted-space"><span lang="EN-US"> </span></span><span lang="EN-US">mpiwg-sessions <<a href="mailto:mpiwg-sessions-bounces@lists.mpi-forum.org"><span style="color:#467886">mpiwg-sessions-bounces@lists.mpi-forum.org</span></a>><span class="apple-converted-space"> </span><b>On
Behalf Of<span class="apple-converted-space"> </span></b>Holmes, Daniel John via mpiwg-sessions<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Monday, January 29, 2024 4:13 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>MPI Sessions working group <<a href="mailto:mpiwg-sessions@lists.mpi-forum.org"><span style="color:#467886">mpiwg-sessions@lists.mpi-forum.org</span></a>>; Aurelien Bouteiller <<a href="mailto:bouteill@icl.utk.edu"><span style="color:#467886">bouteill@icl.utk.edu</span></a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span>Holmes, Daniel John <<a href="mailto:daniel.john.holmes@intel.com"><span style="color:#467886">daniel.john.holmes@intel.com</span></a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [mpiwg-sessions] Sessions WG - meet 1/29/24</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Hi Howard/all,<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Here is the simple code I was talking about in the meeting today:<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">// general high-level optimistic application</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">void main() {</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Session session;</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Session_Init(MPI_INFO_NULL, MPI_ERRORS_ARE_FATAL, &session);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Group group;</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Group_from_session_pset(session, "<a href="mpi://world"><span style="color:#467886">mpi://world</span></a>", &group);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Comm comm;</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Comm_create_from_group(group, &comm);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> ret = do_stuff_with_comm(comm);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> if (MPI_SUCCESS == ret) {</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Comm_disconnect(&comm);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Session_Finalize(&session);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> break;</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> } else {</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> panic();</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">}</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">// general high-level pragmatic application</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">void main() {</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> // additional code</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> while (1) {</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Session session;</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Session_Init(MPI_INFO_NULL, MPI_ERRORS_RETURN, &session);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Group world, failed, group;</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Group_from_session_pset(session, "<a href="mpi://world"><span style="color:#467886">mpi://world</span></a>", &world);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> // additional code</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Session_get_proc_failed(session, &failed); // new API, seems easy to do</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Group_difference(world, failed, &group);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Group_free(&world);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Group_free(&failed);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Comm comm;</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Comm_create_from_group(group, &comm); // <-- the detail-devils live here</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Group_free(&group);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> ret = do_stuff_with_comm(comm);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Comm_disconnect(&comm);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> MPI_Session_Finalize(&session);</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> if (MPI_SUCCESS == ret) {</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> break; // all done!</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> } else if (MPI_ERR_PROC_FAILED == ret) {</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> continue; // no more panic</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> }</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> // additional code</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New""> } // end while</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">}</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Best wishes,<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">Dan.<span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-color:currentcolor currentcolor;border-image: none">
<div>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span class="apple-converted-space"><span lang="EN-US"> </span></span><span lang="EN-US">mpiwg-sessions <<a href="mailto:mpiwg-sessions-bounces@lists.mpi-forum.org"><span style="color:#467886">mpiwg-sessions-bounces@lists.mpi-forum.org</span></a>><span class="apple-converted-space"> </span><b>On
Behalf Of<span class="apple-converted-space"> </span></b>Pritchard Jr., Howard via mpiwg-sessions<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Thursday, January 25, 2024 6:07 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>MPI Sessions working group <<a href="mailto:mpiwg-sessions@lists.mpi-forum.org"><span style="color:#467886">mpiwg-sessions@lists.mpi-forum.org</span></a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span>Pritchard Jr., Howard <<a href="mailto:howardp@lanl.gov"><span style="color:#467886">howardp@lanl.gov</span></a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>[mpiwg-sessions] Sessions WG - meet 1/29/24</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",sans-serif"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif">Hi Folks,</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif">Let’s meet on 1/29 to continue discussions related to sessions and FT. </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif">I think what will help is to consider several use cases and implications.</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif">Here are some I have</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-top:0cm;margin-bottom:0cm;mso-list:l0 level1 lfo1">
<span lang="EN-US" style="font-family:"Aptos",sans-serif">App using sessions to init/finalize and create at least one initial communicator with MPI_Comm_create_from_group, but also wants to use methods available in slice1 of ULFM proposal to shrink/repair
communicators. Are there any problems?</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin-top:0cm;margin-bottom:0cm;mso-list:l0 level1 lfo1">
<span lang="EN-US" style="font-family:"Aptos",sans-serif">App using sessions to init/finalize and create at least one initial communicator with MPI_Comm_create_from_group, and wants to use methods available in slice 1 of ULFM proposal to create new group from
a pset and create a new communicator</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin-top:0cm;margin-bottom:0cm;mso-list:l0 level1 lfo1">
<span lang="EN-US" style="font-family:"Aptos",sans-serif">App using sessions to init/finalize, etc. and when a fail-stop error is detected destroy the session, create a new session query for process sets, etc. and start all over.<span class="apple-converted-space"> </span></span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></li></ul>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif">We should also consider the behavior of MPI_Comm_create_from_group if a process failure occurs while creating a new communicator. The ULFM slice 1 discusses behavior of MPI_COMM_DUP
and process failure. We’d probably want similar behavior for MPI_Comm_create_from_group.</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif">For those with access, the ULFM slice 1 PR is at<span class="apple-converted-space"> </span><a href="https://github.com/mpi-forum/mpi-standard/pull/947"><span style="color:#467886">https://github.com/mpi-forum/mpi-standard/pull/947</span></a></span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif">Thanks,</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif">Howard</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">-------</span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="366" style="width:274.5pt;border-collapse:collapse">
<tbody>
<tr style="height:129.6pt">
<td width="76" valign="top" style="width:56.95pt;padding:0cm 5.4pt 0cm 5.4pt;height:129.6pt">
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif"><image001.png></span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
</td>
<td width="290" valign="top" style="width:217.55pt;padding:0cm 5.4pt 0cm 5.4pt;height:129.6pt">
<div>
<p class="MsoNormal"><b><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:#0B1A8C">Howard Pritchard</span></b><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#545961">Research Scientist</span></b><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#545961">HPC-ENV</span></b><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial",sans-serif"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C">Los Alamos National Laboratory</span></b><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C"><a href="mailto:howardp@lanl.gov"><span style="color:#0563C1">howardp@lanl.gov</span></a></span></b><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C"> </span></b><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Aptos",sans-serif"><a href="https://www.instagram.com/losalamosnatlab/"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;text-decoration:none"><image002.png></span></b></a><a href="https://twitter.com/LosAlamosNatLab"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;text-decoration:none"><image003.png></span></b></a><a href="https://www.linkedin.com/company/los-alamos-national-laboratory/"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;text-decoration:none"><image004.png></span></b></a><a href="https://www.facebook.com/LosAlamosNationalLab/"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#0B1A8C;text-decoration:none"><image005.png></span></b></a><o:p></o:p></span></p>
</div>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Aptos",sans-serif"> </span><span style="font-family:"Aptos",sans-serif"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif">_______________________________________________<br>
mpiwg-sessions mailing list<br>
</span><a href="mailto:mpiwg-sessions@lists.mpi-forum.org"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#467886">mpiwg-sessions@lists.mpi-forum.org</span></a><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif"><br>
</span><a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:#467886">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</span></a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>