<div dir="ltr"><div>The question does essentially boil down to whether a full fan-out of nonblocking send/recv pairs followed by wait-all is a valid implementation of MPI_Barrier. Reviewing the text that Dan cited for MPI 4.0:<br></div><div><br></div><div><div>> (§5.14, p234 MPI-2019-draft): “A correct, portable program must invoke collective communications so that deadlock will not occur”</div></div><div><br></div><div>There isn't any convenient way the user can find out about remote completion of a barrier (short of building their own barrier with synchronous send). So, we can either interpret the above statement to place a strong completion requirement on collectives (bad for performance). Or, we can interpret it to mean that there's really no safe time when a user can call into a blocking external interface. The RMA progress passage that Martin referenced seems to support this latter interpretation with the sockets example given in the rationale.</div><div><br></div><div> ~Jim.</div><div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 12, 2020 at 11:17 AM HOLMES Daniel <<a href="mailto:d.holmes@epcc.ed.ac.uk">d.holmes@epcc.ed.ac.uk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
Hi Jim, et al,
<div><br>
</div>
<div>
<div>Unless the point-to-point pseudo-code given is proven to be a valid implementation of MPI Barrier, then reasoning about MPI Barrier using it as a basis is unlikely to be edifying.</div>
</div>
<div>I also have a (possibly flawed) implementation of MPI Barrier that exhibits some odd semantics/behaviours and I could use that to assert (likely incorrectly) that MPI Barrier is defined in a way that exhibits those semantics/behaviours. However,
that serves no purpose, so I won’t dwell on it any further.</div>
<div><br>
</div>
<div>
<div>I’m glad that someone responded with a reference to the MPI Standard, thanks Martin. In that vein, here’s my tuppence:</div>
<div><br>
</div>
</div>
<div>The definition of MPI Barrier in the MPI Standard states (§5.3, p149 in MPI-2019-draft):</div>
<div>“If comm is an intracommunicator, MPI_BARRIER blocks the caller until all group members have called it. The call returns at any process only after all group members have entered the call.”</div>
<div><br>
</div>
<div>There is a happens-before between “all MPI processes have entered the MPI Barrier call” and “MPI processes are permitted to leave the call”. That’s it; that’s all MPI Barrier does/is required to ensure.</div>
<div><br>
</div>
<div>There is no indication or requirement for alacrity. This appears to be a valid (although stupid) implementation:</div>
<div>int MPI_Barrier(MPI_comm comm) {</div>
<div> int ret = PMPI_Barrier(comm);</div>
<div> sleep(100days);</div>
<div> return ret;</div>
<div>}</div>
<div><br>
</div>
<div>There is no indication or suggestion for how or when MPI processes become aware that the necessary pre-condition for returning control to the user has been satisfied. Some may become aware of this situation a significant amount of time before/after
others. Local completion does not guarantee remote completion in MPI (except for passive-target RMA, e.g. MPI_Win_unlock).</div>
<div><br>
</div>
<div>There is no indication or requirement that the necessary pre-condition is also a sufficient pre-condition, although we may wish to assume that and we may wish to clarify the wording of the MPI Standard to specify that explicitly. If the MPI Standard
text were changed to “The call returns at any process <strike>only</strike>immediately after all group members have entered the call.” then (given the other usage of immediately in the MPI Standard) we could assume that the procedure becomes strong local (immediate)
once the necessary pre-condition is met. Without the word “immediate” in the sentence, the return of the MPI procedure is permitted to require remote progress, i.e. after the necessary pre-condition is met, it becomes weak local (called local in the MPI Standard).
Some MPI libraries (can, if configured in a particular way) provide strong progress; however, MPI only requires weak progress. Weak progress means it is permitted for remote progress to happen only during remote MPI procedure calls.</div>
<div><br>
</div>
<div>So,</div>
<div><br>
</div>
<div>If MPI required “returns immediately after...” (which it does not) then every MPI process would be required to ensure the remote completion of its “send” (as well as local completion of the “recv”) before it returns control to the user. This would
mean that our intuitive feel for what MPI_Barrier should do would be correct and the suggested point-to-point code would be an incorrect implementation of MPI_Barrier.</div>
<div>If MPI required strong progress (which it does not) then every MPI process would eventually become aware that it is permitted to return control to the user, without additional remote MPI procedure calls. This would mean that our intuitive feel
for what MPI_Barrier should do would be correct and the suggested point-to-point code would be a correct implementation of MPI_Barrier.</div>
<div><br>
</div>
<div>As it is, our intuitive feel for what MPI_Barrier should do is probably wrong (i.e. not what MPI actually specifies), or at least too optimistic because it depends on a high quality implementation that exceeds what is minimally specified by the
MPI Standard as required.</div>
<div>As it is, the MPI_Barrier in the original question does not guard against problems with the non-MPI file operations - indeed, adding it introduces a new possibility of a deadlock, which would not be present in the code without the MPI_Barrier
operation.</div>
<div><br>
</div>
<div>I would argue that the original code is therefore erroneous (incorrect/non-portable) because (§5.14, p234 MPI-2019-draft):</div>
<div>“A correct, portable program must invoke collective communications so that deadlock will not occur”</div>
<div><br>
</div>
<div>One correct program that achieves what the original looks like it might be trying to achieve (IHMO) is as follows:</div>
<div>if (rank == 1)<br>
create_file("test”);<br>
MPI_Barrier();</div>
<div>if (rank == 0)<br>
<div> while not_exists("test")</div>
sleep(1);</div>
<div>This program still assumes that the file creation actually creates the file and flushes it to a filesystem that makes it visible to the existence check but that must be true if the code-without-MPI is correct, i.e. adding MPI has not introduced
a new problem to the code.</div>
<div><br>
</div>
<div>Taking this reasoning about the minimal requirements of MPI Barrier (at least) one step too far, the only restriction on implementation of MPI_Barrier seems to be “do not return until <something happens>”, which suggests this is a valid (although
very unhelpful) implementation:</div>
<div>
<div>int MPI_Barrier(MPI_comm comm) {</div>
<div> while (1); // do not return, ever</div>
<div>}</div>
</div>
<div><br>
</div>
<div>To guard against low-quality/malicious implementations of the MPI Standard, we could either clarify the wording of the text about MPI_Barrier (and probably the text about every other MPI procedure) to include the concept of becoming an “immediate”
procedure once certain criteria are met (likely to be a lot of effort/angst for some), or mandate strong progress for all MPI libraries (likely to be very unpopular for some).</div>
<div><br>
</div>
<div>
<div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Cheers,</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Dan.</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
—<br>
Dr Daniel Holmes PhD<br>
Architect (HPC Research)<br>
<a href="mailto:d.holmes@epcc.ed.ac.uk" target="_blank">d.holmes@epcc.ed.ac.uk</a><br>
Phone: +44 (0) 131 651 3465<br>
Mobile: +44 (0) 7940 524 088<br>
Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
—</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
—</div>
</div>
</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On 12 Oct 2020, at 10:04, Martin Schulz via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank">mpi-forum@lists.mpi-forum.org</a>> wrote:</div>
<br>
<div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-US">Hi Jim, all,<u></u><u></u></span></div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-US"><u></u> <u></u></span></div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-US">We had a similar discussion (in a smaller circle) during the terms discussions – at least to my understanding, all bets are off as soon as you add dependencies and wait conditions outside of MPI, like here with the file. A note to
this point is in a rational (Section 11.7, page 491 in the 2019 draft) – based on that an MPI implementation is allowed to deadlock (or cause a deadlock) – if all dependencies would be in MPI calls, then “eventual” progress should be guaranteed – even if it
is after the 100 days in Rajeev’s example: that would – as far as I understand – still be correct behavior, as no MPI call is guaranteed to return in a fixed finite time (all calls are at best “weak local”).<u></u><u></u></span></div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-US"><u></u> <u></u></span></div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-US">Martin<u></u><u></u></span></div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-US"><u></u> <u></u></span></div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-US"><u></u> <u></u></span></div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-US">-- </span><span lang="EN-US" style="font-size:10.5pt;font-family:Helvetica"><br>
Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems<br>
Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching<br>
Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ)<br>
Email:<span> </span><a href="mailto:schulzm@in.tum.de" style="color:blue;text-decoration:underline" target="_blank">schulzm@in.tum.de</a></span><span lang="EN-US"><u></u><u></u></span></div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span lang="EN-US"><u></u> <u></u></span></div>
</div>
</div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(181,196,223);padding:3pt 0cm 0cm">
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<b><span style="font-size:12pt">From:<span> </span></span></b><span style="font-size:12pt">mpi-forum <<a href="mailto:mpi-forum-bounces@lists.mpi-forum.org" target="_blank">mpi-forum-bounces@lists.mpi-forum.org</a>>
on behalf of Jim Dinan via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank">mpi-forum@lists.mpi-forum.org</a>><br>
<b>Reply-To:<span> </span></b>Main MPI Forum mailing list <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank">mpi-forum@lists.mpi-forum.org</a>><br>
<b>Date:<span> </span></b>Sunday, 11. October 2020 at 23:41<br>
<b>To:<span> </span></b>"Skjellum, Anthony" <<a href="mailto:Tony-Skjellum@utc.edu" target="_blank">Tony-Skjellum@utc.edu</a>><br>
<b>Cc:<span> </span></b>Jim Dinan <<a href="mailto:james.dinan@gmail.com" target="_blank">james.dinan@gmail.com</a>>, Main MPI Forum mailing list <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank">mpi-forum@lists.mpi-forum.org</a>><br>
<b>Subject:<span> </span></b>Re: [Mpi-forum] [EXT]: Progress Question<u></u><u></u></span></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
You can have a situation where the isend/irecv pair completes at process 0 before process 1 has called irecv or waitall. Since process 0 is now busy waiting on the file, it will not make progress on MPI calls and can result in deadlock.<span> </span><u></u><u></u></div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
~Jim.<u></u><u></u></div>
</div>
</div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
<div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
On Sat, Oct 10, 2020 at 2:17 PM Skjellum, Anthony <<a href="mailto:Tony-Skjellum@utc.edu" style="color:blue;text-decoration:underline" target="_blank">Tony-Skjellum@utc.edu</a>> wrote:<u></u><u></u></div>
</div>
<blockquote style="border-style:none none none solid;border-left-width:1pt;border-left-color:rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm" type="cite">
<div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span style="font-size:12pt">Jim, OK, my attempt at answering below.<u></u><u></u></span></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span style="font-size:12pt"><u></u> <u></u></span></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span style="font-size:12pt">See if you agree with my annotations.<u></u><u></u></span></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span style="font-size:12pt"><u></u> <u></u></span></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span style="font-size:12pt">-Tony<u></u><u></u></span></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span style="font-size:12pt"><u></u> <u></u></span></div>
</div>
<div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span style="font-size:12pt"><u></u> <u></u></span></div>
</div>
<div id="gmail-m_-1612759991888082496gmail-m_-4671823654087486545Signature">
<div>
<div id="gmail-m_-1612759991888082496gmail-m_-4671823654087486545divtagdefaultwrapper">
<div style="margin:0cm"><span style="font-size:12pt">Anthony Skjellum, PhD<u></u><u></u></span></div>
<div style="margin:0cm"><span style="font-size:12pt">Professor of Computer Science and Chair of Excellence<u></u><u></u></span></div>
<div style="margin:0cm"><span style="font-size:12pt">Director, SimCenter<u></u><u></u></span></div>
<div style="margin:0cm"><span style="font-size:12pt">University of Tennessee at Chattanooga (UTC)<u></u><u></u></span></div>
<div style="margin:0cm"><span style="font-size:12pt"><a href="mailto:tony-skjellum@utc.edu" style="color:blue;text-decoration:underline" target="_blank">tony-skjellum@utc.edu</a><span> </span> [or<span> </span><a href="mailto:skjellum@gmail.com" style="color:blue;text-decoration:underline" target="_blank">skjellum@gmail.com</a>]<u></u><u></u></span></div>
<div style="margin:0cm"><span style="font-size:12pt">cell: 205-807-4968<u></u><u></u></span></div>
<div style="margin:0cm"><span style="font-size:12pt"><u></u> <u></u></span></div>
</div>
</div>
</div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span style="font-size:12pt"><u></u> <u></u></span></div>
</div>
<div class="MsoNormal" align="center" style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;text-align:center">
<hr size="0" width="95%" align="center">
</div>
<div id="gmail-m_-1612759991888082496gmail-m_-4671823654087486545divRplyFwdMsg">
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<b><span>From:</span></b><span><span> </span>mpi-forum <<a href="mailto:mpi-forum-bounces@lists.mpi-forum.org" style="color:blue;text-decoration:underline" target="_blank">mpi-forum-bounces@lists.mpi-forum.org</a>>
on behalf of Jim Dinan via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" style="color:blue;text-decoration:underline" target="_blank">mpi-forum@lists.mpi-forum.org</a>><br>
<b>Sent:</b><span> </span>Saturday, October 10, 2020 1:31 PM<br>
<b>To:</b><span> </span>Main MPI Forum mailing list <<a href="mailto:mpi-forum@lists.mpi-forum.org" style="color:blue;text-decoration:underline" target="_blank">mpi-forum@lists.mpi-forum.org</a>><br>
<b>Cc:</b><span> </span>Jim Dinan <<a href="mailto:james.dinan@gmail.com" style="color:blue;text-decoration:underline" target="_blank">james.dinan@gmail.com</a>><br>
<b>Subject:</b><span> </span>[EXT]: [Mpi-forum] Progress Question</span><u></u><u></u></div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u><u></u></div>
</div>
</div>
<div>
<table border="0" cellpadding="0" width="100%" id="gmail-m_-1612759991888082496gmail-m_-4671823654087486545x_header-notice" style="width:1029.61px;background-color:rgb(253,183,54)">
<tbody>
<tr>
<td style="padding:0.75pt">
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;text-align:center">
<strong><span style="font-size:13.5pt;font-family:Calibri,sans-serif;color:rgb(17,46,81)">External Email</span></strong><span style="font-size:13.5pt;color:rgb(17,46,81)"><u></u><u></u></span></div>
</td>
</tr>
</tbody>
</table>
<div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<span style="color:white">Hi All,<span> </span></span><u></u><u></u></div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
A colleague recently asked a question that I wasn't able to answer definitively. Is the following code guaranteed to make progress?<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
<blockquote style="margin-left:30pt;margin-right:0cm" type="cite">
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
MPI_Barrier();<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
-- everything is uncertain to within one message, if layered on pt2pt;<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
--- let's assume a power of 2, and recursive doubling (RD).<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
--- At each stage, it posts an irecv and isend to its corresponding element in RD<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
--- All stages must complete to get to the last stage.<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
--- At the last stage, it appears like your example below for N/2 independent process pairs, which appears always to complete.<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
Oif rank == 1<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
create_file("test")<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
if rank == 0<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
while not_exists("test")<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
sleep(1);<u></u><u></u></div>
</div>
</blockquote>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
That is, can rank 1 require rank 0 to make MPI calls after its return from the barrier, in order for rank 1 to complete the barrier? If the code were written as follows:<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
<blockquote style="margin-left:30pt;margin-right:0cm" type="cite">
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
isend(..., other_rank, &req[0])<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
irecv(..., other_rank, &req[1])<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
waitall(2, req)<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
--- Assume both isends buffer on the send-side and return immediately--valid.<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
--- Both irecvs are posted, but unmatched as yet. Nothing has transferred on network.<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
--- Waitall would mark the isends done at once, and work to complete the irecvs; in<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
that process, each would have to progress the isends across the network. On this comm<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
and all comms, incidentally. <u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
--- When waitall returns, the data has transferred to the receiver, otherwise the irecvs <u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
aren't done.<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
if rank == 1<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
create_file("test")<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
if rank == 0<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
while not_exists("test")<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
sleep(1);<u></u><u></u></div>
</div>
</blockquote>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
I think it would clearly not guarantee progress since the send data can be buffered. Is the same true for barrier?<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
Cheers,<u></u><u></u></div>
</div>
<div>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
~Jim.<u></u><u></u></div>
</div>
</div>
</div>
<table border="0" cellpadding="0" width="100%" id="gmail-m_-1612759991888082496gmail-m_-4671823654087486545x_footer-notice" style="width:1029.61px;background-color:rgb(253,183,54)">
<tbody>
<tr>
<td style="padding:0.75pt">
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif;text-align:center">
<strong><span style="font-size:13.5pt;font-family:Calibri,sans-serif;color:rgb(17,46,81)">This message is not from a<span> </span><a href="http://utc.edu/" style="color:blue;text-decoration:underline" target="_blank">UTC.EDU</a><span> </span>address.
Caution should be used in clicking links and downloading attachments from unknown senders or unexpected email.<span> </span></span></strong><span style="font-size:13.5pt;color:rgb(17,46,81)"><u></u><u></u></span></div>
</td>
</tr>
</tbody>
</table>
<div style="margin:0cm;font-size:11pt;font-family:Calibri,sans-serif">
<u></u> <u></u></div>
</div>
</div>
</blockquote>
</div>
</div>
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">_______________________________________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">mpi-forum
mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline"><a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank">mpi-forum@lists.mpi-forum.org</a></span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline"><a href="https://lists.mpi-forum.org/mailman/listinfo/mpi-forum" target="_blank">https://lists.mpi-forum.org/mailman/listinfo/mpi-forum</a></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote></div>