<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Ralph,
<div class=""><br class="">
</div>
<div class="">In general, I agree with all of this. Two points:</div>
<div class=""><br class="">
</div>
<div class="">1) I think your assignment of special case and general case are the wrong way around. A fully connected graph (your normal/general case) is a special case of a general graph (your special case), which may be completely disconnected, partially
connected, disjointly connected, or fully connected. Requiring every process to specify every other process (your normal/general case) is therefore restricting their behaviour from the general graph (your special case) to a particular special case. This does
not need to be an issue: generalising a special-case API does not break backwards compatibility, in general. On the other hand, I’m not sure how much different either the specification’s wording or the implementation’s code would have to be in order to support
the general graph case as well as the fully connected special case. The general case definitely messes with the brain cells a lot more though.</div>
<div class=""><br class="">
</div>
<div class="">2a) Having a single leader may not be sufficient. I’m thinking intercommunicators, which have naturally have two leaders - one for the local group and one for the remote group. This statement immediately gives us a clue as to how to handle that
particular use-case: group construct two separate groups then worry about linking them together later. Perhaps, a third group containing just the two leaders? OK, maybe that works fine as is. Taking a different tack, unless the leader treats no news as good
news, all processes must notify the leader that they believe the operation is successful. For a really large group, wouldn’t the n-to-1 ‘notify the leader of local outcome’ gather/reduce operation cause a bottleneck at the leader? You’d want a hierarchy (e.g.
a tree or something like it) just like for normal gather/reduce implementations, i.e. different processes would perceive different leaders (who in turn may know they are just middle-management and there is a big boss that their underlings don’t know about).
Failure of a leaf would be notified to their parent/leader process, which can fix/swallow the problem or bubble it up to their parent/leader. Failure of a leader causes an election in their sub-tree(s)/children. This gives the possibility of a free choice
of the size of the containment domain - preferable to being told it’s all or nothing. A general graph naturally gives a structure but it would be hard, in general, to use that for containment - because each process can specify more than one other process,
which is their leader? Thus, each process could nominate a single process (probably one of the entries in their process array, but not necessarily) to be their leader, or a rule could be devised that figures out which process is leader for each process.</div>
<div class="">
<div class=""><br class="webkit-block-placeholder">
</div>
<div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
2b) Nominating/electing a (successor) leader could be a simple as “proceed in index order” (of the locally provided array). PMIx taking over by self-nominating leadership and canceling the group construct at the first sign of trouble seems a bit harsh.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Conceptually, I think we can take your 1-to-n model, replicate it m times (possibly with different values of n in each replica), then add one more process that is the leader of all the m “leaders”. Rinse and repeat to obtain any tree-based hierarchy (partway
towards the general graph case). Now draw in “connections” from each process to all processes in their local array (which might be all other processes in the group, in the fully connected special case) and we have the general graph case. The tree/hierarchy
defines the flow for control messages, the arrays give the final connectivity topology.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Calling this proposed function with the intention of creating a fully connected group (your normal case), requires each process to specify an array of processes (supported by the current API) containing all other processes. The first process in the array is
nominated leader - PMIx sends termination notices there. If it fails, then the process at index (last_leader_index +1) is nominated until there are no more processes to nominate, at which point PMIx self-nominates and cancels the operation. If a non-leader
fails, its current leader is notified. That process decides what to do. A cancel decision is broadcast to all other processes, an group join invitation is send to the invited process, a "don’t care, carry on” is silent. When a non-leader joins the group, its
leader is notified. When the leader has responses from all its non-leaders, it declares the group complete, which is broadcast to all non-leader processes.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Calling this proposed function with the intention of creating a dist_graph (your special case), requires each process to specify an array of processes (supported by the current API) containing only its neighbour processes. The implementation sketched above
is the same for this case. A middle-manager plays the leader role to some processes and the non-leader role to exactly one other process (at a time).</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
In both use-cases, all processes must register for leader events and non-leader event, because they can be nominated/promoted at any time during the group construct by PMIx depending on what happens.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="Apple-interchange-newline">
Cheers,</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Dan.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Applications Consultant in HPC Research<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
—</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
—</div>
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 9 Aug 2018, at 16:42, Ralph H Castain <<a href="mailto:rhc@open-mpi.org" class="">rhc@open-mpi.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="Apple-interchange-newline">
<br class="">
<blockquote type="cite" class="">
<div class="">On Aug 9, 2018, at 7:53 AM, HOLMES Daniel <<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
Hi Ralph,
<div class=""><br class="">
</div>
<div class="">I particularly like the self-fulfilling paradox at the end there: if the group construct fails then I won’t even call group construct, which guarantees that it’ll fail. Of course it’s not really a paradox because there are multiple processes involved
but I can see why your head is hurting.<br class="">
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Yes, it is easy to rapidly get trapped in the weeds here.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="">
<div class=""><br class="webkit-block-placeholder">
</div>
<div class="">Taking a further step back, scalable group construction (without a priori knowledge of membership and with the possibility of errors) requires a distributed graph specification of the membership and handling of errors. If each process only specifies
its neighbours, and only handles errors from its neighbours, then the event explosion can be contained to a neighbourhood.</div>
<div class=""><br class="">
</div>
<div class="">For example, if all processes specified the group membership as {back_proc, forward_proc} then a failed process will cause a NOTIFY_TERMINATION event in its back_proc and its forward_proc but no others, irrespective of the total size of the intended
group. Those two processes must then decide what to do, e.g. link to each other (form a ring with one fewer processes than originally intended) or invite a new process (hopefully the same one ;) which will fix the originally intended ring). A “burn the world”
decision from either of these two processes must then be propagated to all other participants (not necessarily around the ring, an OOB method works too). None of the other processes can have left/completed their group construct yet because not all members
have given their consent (implies: connect up local neighbourhood, wait for go signal, complete locally).</div>
<div class=""><br class="webkit-block-placeholder">
</div>
<div class="">
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
If the user chooses to specify all other processes at all processes, then they are explicitly requesting to be notified about all events for which they have registered from all other processes. This may cause explosions for large groups.</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
<br class="">
</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
Note, the user does not actually have to specify the intended size of the group when constructing it. The size and topology of the (connected portion of the) graph can be determined once the group exists, by doing collective operations using the group. Collective
operations can be done on the entire (connected portion of the) graph by each process interacting only with its neighbours. Having discovered information about processes outside of one’s local neighbourhood, direct communication can be done between any pair
or sub-group of processes. Thus, a fully connected MPI communicator can (if required) be built up from a MPI_DIST_GRAPH topology specification.</div>
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
I’m reluctant to customize PMIx Groups for just the graph use-case as other programming models also have interest (and might not fit that case) and it isn’t clear to me that everyone in MPI will want to base themselves on a graph-based group approach . What
we could do, though, is provide attributes to support the use model you describe and let the PMIx Group implementation deal with it as a special case.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
For the more general case, I think we just need to ensure that there is a clean failure path that ensures the user gets out of the operation (i.e., doesn’t hang or incorrectly think the group exists) when failures occur. We can provide failure notification
and recovery methods - we just need to acknowledge that these only really work in the (expected) case where failures are relatively rare events. After all, if lots of processes are failing or refusing to join the proposed group during a construct operation,
then you probably need to do some triage on your cluster and/or your application!</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
If we take that approach, then we can limit notifications to the “leader” and let it decide what to do about it. If the leader fails, then we could just have PMIx automatically terminate group construction, issuing “cancel” events to all other participants.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
For flexibility, we can add an attribute that modifies that behavior and add a new event to notify other group participants of the leader’s failure (we know the leader already agreed to join the group!). We can then add an attribute by which a process can declare
itself the new leader, thereby causing an event to the rest of the group participants to update their leader assignment (this is implemented today as a broadcast and so scales relatively well). The new leader is the one that will decide what to do about giving
up on constructing the group.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
Since we cache notifications, we know that any “cancel” event received by a proc prior to registering for it will still be delivered. We then specify in the standard that procs should register for all group-related events prior to engaging in any PMIx Group
operations. This ensures that the app knows about the “cancel” before calling construct, and that procs which call the blocking form of construct prior to the event arriving will still have a mechanism for getting out of the operation.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
To make the registration easier, PMIx could add an ability to register for a “class” of events - e.g., register for the “group” class of events. This would provide for future compatibility should new group-related events get added. You currently have to specify
the events you want to know about.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
Make sense?</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
Ralph</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="">
<div class="">
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
<br class="Apple-interchange-newline">
Cheers,</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
Dan.</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Applications Consultant in HPC Research<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—</div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On 9 Aug 2018, at 15:00, Ralph H Castain <<a href="mailto:rhc@open-mpi.org" class="">rhc@open-mpi.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space;">So let’s step back for a moment at look at the failed construct problem in more depth. There are a couple of issues that come to mind.
<div class=""><br class="">
</div>
<div class="">First, do we really want to send the NOTIFY_TERMINATION event to all participants, or just to the leader? If the group under construction is large and we see a number of failures, then we could wind up in an event “storm”. If we alert only the
leader, then it begs the question: what if the leader is the one who fails? Do we need a mechanism by which someone else can declare themselves to be the “leader”?</div>
<div class=""><br class="">
</div>
<div class="">It isn’t too difficult for us to examine the returned results array from an event handler, though I’d want to (a) generalize it a bit so we can (b) limit how much of that we do to avoid making the event notification code explode with special cases.
If we go that route, which seems the right thing to me, then.we again have a couple of choices:</div>
<div class=""><br class="">
</div>
<div class="">* if the NOTIFY_TERMINATION event is only going to the leader, then we would provide the ability for the leader to declare “burn the world” and send a corresponding event to all participants. It does create a bit of a race condition as a remote
participating proc may get the event prior to calling Group_join and thus (a) has no idea what the event is talking about and (b) would have to retain the cancellation notice pending the call to Group_join so it could return an error. Doable - just a tad tricky
and difficult to test that race condition</div>
<div class=""><br class="">
</div>
<div class="">* if the event goes to all participants, then they could locally decide to abandon the group. If they have already joined, they could leave. If not, then they could simply decline the invite. Again, there are race conditions that could bite us
(particularly in multi-threaded apps), but maybe we resolve some of those by imposing requirements on the app.</div>
<div class=""><br class="">
</div>
<div class="">Now that was all based on the async construct - but what do we do about a blocking call to PMIx_Group_construct? Only think I can think of would be to provide an attribute in the results array that tells the PMIx library to “kick me out of the
current operation” and includes some tag(s) to indicate what operation it is talking about. We actually talked about that at some length during the last in-person PMIx devel meeting and came up with a scheme to support such a request (hasn’t been implemented
yet), so this could work. However, it again creates that race condition for procs that receive the TERMINATION event prior to calling “construct” as the operation hasn’t been initiated yet. I guess we could just put the burden on the app to realize that it
got a group_construct termination event and should therefore not call “construct” on that group?</div>
<div class=""><br class="">
</div>
<div class="">My head is beginning to hurt and I’ve probably confused folks anyway, so best to stop here and wait for input.</div>
<div class="">Ralph</div>
<div class=""><br class="">
</div>
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 9, 2018, at 2:26 AM, HOLMES Daniel <<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
Hi Ralph,
<div class=""><br class="">
</div>
<div class="">Thanks for the updates, I’ll take a look in a moment (after coffee).</div>
<div class=""><br class="">
</div>
<div class="">Technically, a cancel API in PMIX *could* be used with a blocking group construction by calling the cancel inside an event callback. For example, an app could keep a count of how many times PMIX_GROUP_NOTIFY_TERMINATION was called for this group
construction and invite a new process for the first N times but cancel the operation in response to any subsequent event(s).</div>
<div class="">
<div class=""><br class="webkit-block-placeholder">
</div>
<div class="">
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
<div class="">MPI is trying to get rid of its cancel API - we deprecated cancel for point-to-point send because it is fundamentally broken. However, cancel for point-to-point receive is still valid and useful. The problem with cancel is always the race between
the “all is OK, go ahead” and the “whoa, stop that” signals. With a receive in MPI, the choice of which will succeed (receive or cancel) is always a local decision and can therefore be made atomic and consistent. The choice between send and cancel is a distributed
decision, which cannot be atomic, and always suffers from a race. One way to avoid this race in a PMIX group cancel would be specify that it is only valid from within an event callback, e.g. by exposing it as an in-out/by-ref parameter in the callback itself
(not as a separate function call). PMIX could then examine this parameter (set by the user, during the event callback) when the callback returns. It’s a binary/boolean choice between “I handled it, carry on” and “I panicked, burn the world”.</div>
<div class=""><br class="">
</div>
<div class="">Is it useful? Well all systems have finite resources to use as replacements, so eventually this operation must fail. This type of cancel allows the application to choose when to give up based on how many things happened rather than just how many
seconds elapsed.</div>
<div class=""><br class="">
</div>
Cheers,</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
Dan.</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Applications Consultant in HPC Research<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—</div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On 9 Aug 2018, at 03:54, Ralph H Castain <<a href="mailto:rhc@open-mpi.org" class="">rhc@open-mpi.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space;">I have updated the web page to reflect the comments. Let me know what you think and about the “cancel” API.<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 8, 2018, at 5:40 PM, Ralph H Castain <<a href="mailto:rhc@open-mpi.org" class="">rhc@open-mpi.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space;"><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 8, 2018, at 4:14 PM, HOLMES Daniel <<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
Hi Ralph,
<div class=""><br class="">
</div>
<div class="">I was nodding whilst reading,</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Great!</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="">until I got to PMIX_GROUP_LEAVE_REQUEST (Destruct procedure, PMIx_Groups).</div>
<div class=""><br class="">
</div>
<div class="">There are situations where a process will leave without the luxury of requesting and being patient first, e.g. faults (handled as termination, I know, bear with me a moment). If this event was instead PMIX_GROUP_LEFT, then processes would be written
to be able to cope with sudden exits in other processes. They have to be written like that anyway because of PMIX_GROUP_NOTIFY_TERMINATION. This event simply distinguishes "the process called PMIx_GROUP_LEAVE" from "the RM figured out a process stopped executing
(normally or abnormally)”. Is such a distinction useful?</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
It’s a good point. My thought was to provide a “clean” way of dynamically leaving a group as opposed to just pulling out. On the other hand, we do need apps to be prepared for unexpected termination - so it isn’t clear that there is any real benefit. I have
no issue with making this change.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class=""><br class="">
</div>
<div class="">
<div class="">In terms of outstanding/in-progress collective operations, just state that calling PMIX_GROUP_LEAVE is not allowed unless no such operations are in flight.</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
I think putting a requirement that no collective op can be in progress is unenforceable, especially if you take the position that leaving is the same as unexpected termination - i.e., programs need to be written in a way that can adapt to terminations or departures.
We can provide an event indicating that departure occurred and user apps need to register for it and decide for themselves how to respond if in a user collective. The PMIx server can adjust any ongoing PMIx collective (e.g., PMIx_Fence) without user intervention.
We currently error-out from such operations, but we can provide an attribute to indicate the operation should “self-heal” and proceed to completion.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="">
<div class="">The potential race between a process calling PMIX_GROUP_LEAVE and other process(es) in the group starting a collective operation should not happen in a well-defined program. Also, if PMIX_GROUP_NOTIFY_TERMINATION can state "collective operations
will be adjusted appropriately" then why can’t PMIX_GROUP_LEAVE say that too?</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
No problem - it can certainly do so.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="">
<div class=""><br class="webkit-block-placeholder">
</div>
<div class="">
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
<br class="">
</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
For PMIX_GROUP_JOIN, can the leader process give up creation of the group and somehow tell PMIX to stop trying?</div>
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Sure - it can do so by setting the PMIX_TIMEOUT attribute. We could provide a “cancel” API as well, but that would require that you used the non-blocking form of PMIx_Group_construct as otherwise there would be no way to call it. Would a “cancel” API be of
benefit?</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="">
<div class="">
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
If so, then processes that accepted a join request should be informed that the group is never going to be constructed, i.e. they should stop waiting for the callback/return of the blocking function. Thus, "once the group has been completely constructed” could
be tempered with “or the group construction fails”.</div>
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Agreed - will update.</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div class="">
<div class="">
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
<br class="Apple-interchange-newline">
Cheers,</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
Dan.</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Applications Consultant in HPC Research<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—</div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On 8 Aug 2018, at 23:04, Ralph H Castain <<a href="mailto:rhc@open-mpi.org" class="">rhc@open-mpi.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space;">Hi folks
<div class=""><br class="">
</div>
<div class="">I have updated the PMIx Group web page to capture the discussion of the prior meeting plus some subsequent thoughts:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://pmix.org/pmix-standard/pmix-groups/" class="">https://pmix.org/pmix-standard/pmix-groups/</a></div>
<div class=""><br class="">
</div>
<div class="">I’ll try to put some initial implementation behind it before the meeting, so please feel free to chime up with any thoughts.</div>
<div class="">Ralph</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 6, 2018, at 11:38 AM, Ralph H Castain <<a href="mailto:rhc@open-mpi.org" class="">rhc@open-mpi.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="auto" class="">Looks like I can free some time up this week for groups - will try to update later this week
<div class=""><br class="">
<div class="">Sent from my iPhone</div>
<div class=""><br class="">
On Aug 6, 2018, at 11:05 AM, HOLMES Daniel <<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a>> wrote:to<br class="">
<br class="">
</div>
<blockquote type="cite" class="">
<div class="">Hi all,
<div class=""><br class="">
</div>
<div class="">The next meeting for the Sessions WG will be *Tuesday 14th Aug 2018* at 12pm Eastern US time.</div>
<div class=""><br class="">
</div>
<div class="">Note the change of day and time. This is a one-off change due to vacation time.</div>
<div class=""><br class="">
</div>
<div class="">The connection details for the call will be sent out on this list nearer the time.</div>
<div class=""></div>
<div class="">
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
<br class="">
</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
Cheers,</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
Dan.</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Applications Consultant in HPC Research<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space;">
—</div>
</div>
<br class="">
</div>
</blockquote>
<blockquote type="cite" class="">
<div class=""><span class="">The University of Edinburgh is a charitable body, registered in</span><br class="">
<span class="">Scotland, with registration number SC005336.</span><br class="">
</div>
</blockquote>
<blockquote type="cite" class="">
<div class=""><span class="">_______________________________________________</span><br class="">
<span class="">mpiwg-sessions mailing list</span><br class="">
<span class=""><a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a></span><br class="">
<span class=""><a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a></span><br class="">
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
mpiwg-sessions mailing list<br class="">
<a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
The University of Edinburgh is a charitable body, registered in<br class="">
Scotland, with registration number SC005336.<br class="">
_______________________________________________<br class="">
mpiwg-sessions mailing list<br class="">
<a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">
mpiwg-sessions mailing list<br class="">
<a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
_______________________________________________<br class="">
mpiwg-sessions mailing list<br class="">
<a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
The University of Edinburgh is a charitable body, registered in<br class="">
Scotland, with registration number SC005336.<br class="">
_______________________________________________<br class="">
mpiwg-sessions mailing list<br class="">
<a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
mpiwg-sessions mailing list<br class="">
<a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
The University of Edinburgh is a charitable body, registered in<br class="">
Scotland, with registration number SC005336.<br class="">
_______________________________________________<br class="">
mpiwg-sessions mailing list<br class="">
<a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a><br class="">
</div>
</blockquote>
</div>
<br class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">mpiwg-sessions
mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="mailto:mpiwg-sessions@lists.mpi-forum.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">mpiwg-sessions@lists.mpi-forum.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>