<font size=2 face="sans-serif">I have another question about MPI_Comm_agreement:</font>
<br>
<br><font size=2 face="sans-serif">If I want to do this:</font>
<br>
<br><font size=2 face="sans-serif"> ....</font>
<br><font size=2 face="sans-serif"> do
{</font>
<br>
<br><font size=2 face="sans-serif">
if (root) read_some_data_from_a_file(buffer);</font>
<br><font size=2 face="sans-serif">
</font>
<br><font size=2 face="sans-serif">
err = MPI_Bcast(buffer, .... ,root, comm);</font>
<br><font size=2 face="sans-serif">
if (err) {</font>
<br><font size=2 face="sans-serif">
MPI_Comm_invalidate(comm);</font>
<br><font size=2 face="sans-serif">
MPI_Comm_shrink(comm,
&newcomm);</font>
<br><font size=2 face="sans-serif">
MPI_Comm_free(&comm);</font>
<br><font size=2 face="sans-serif">
comm =
newcomm;</font>
<br><font size=2 face="sans-serif">
}</font>
<br>
<br><font size=2 face="sans-serif">
MPI_Comm_size(comm, size);</font>
<br><font size=2 face="sans-serif">
</font>
<br><font size=2 face="sans-serif">
done = do_computation(buffer, size);</font>
<br>
<br><font size=2 face="sans-serif">
/* Let's agree for sure if we are done with
the computation */</font>
<br>
<br><font size=2 face="sans-serif">
MPI_Comm_agreement(comm, &done);</font>
<br>
<br><font size=2 face="sans-serif"> while
(!done);</font>
<br>
<br><font size=2 face="sans-serif"> MPI_Finalize();</font>
<br>
<br><font size=2 face="sans-serif">This code can deadlock because some
ranks may enter MPI_Comm_agreement while others detect an error in MPI_Bcast
and call MPI_Comm_invalidate followed by MPI_Comm_shrink (assume that do_computation
is really, really fast). The call to MPI_Comm_invalidate will not
allow the processes that have already entered MPI_Comm_agreement to leave
that call (P543L45: "</font><font size=2 face="CMTI10">Advice to users.
</font><font size=2 face="CMSS10">MPI</font><font size=2 face="CMTT10">_</font><font size=2 face="CMSS10">COMM</font><font size=2 face="CMTT10">_</font><font size=2 face="CMSS10">AGREEMENT
</font><font size=2 face="CMR10">maintains its collective behavior even</font>
<br><font size=2 face="CMR10">if the </font><font size=2 face="CMSS10">comm
</font><font size=2 face="CMR10">is invalidated. (</font><font size=2 face="CMTI10">End
of advice to users.</font><font size=2 face="CMR10">)" </font><font size=2 face="sans-serif">)
and MPI_Comm_agreement cannot return an error due to the call to MPI_Comm_invalidate
(P545L38: "</font><font size=2 face="CMR10">This function must
not return an error due to process failure (error classes </font><font size=2 face="CMSS10">MPI</font><font size=2 face="CMTT10">_</font><font size=2 face="CMSS10">ERR</font><font size=2 face="CMTT10">_</font><font size=2 face="CMSS10">PROC</font><font size=2 face="CMTT10">_</font><font size=2 face="CMSS10">FAILED
</font><font size=2 face="CMR10">and </font><font size=2 face="CMSS10">MPI</font><font size=2 face="CMTT10">_</font><font size=2 face="CMSS10">ERR</font><font size=2 face="CMTT10">_</font><font size=2 face="CMSS10">INVALIDATED</font><font size=2 face="CMR10">)..."</font><font size=2 face="sans-serif">)
. </font>
<br>
<br><font size=2 face="sans-serif">This would not work:</font>
<br><font size=2 face="sans-serif"> ....</font>
<br><font size=2 face="sans-serif"> do
{</font>
<br>
<br><font size=2 face="sans-serif">
if (root) read_some_data_from_a_file(buffer);</font>
<br><font size=2 face="sans-serif">
</font>
<br><font size=2 face="sans-serif">
err = MPI_Bcast(buffer, .... ,root, comm);</font>
<br><font size=2 face="sans-serif">
if (err) {</font>
<br><font size=2 face="sans-serif">
MPI_Comm_invalidate(comm);</font>
<br><font size=2 face="sans-serif">
MPI_Comm_shrink(comm,
&newcomm);</font>
<br><font size=2 face="sans-serif">
MPI_Comm_free(&comm);</font>
<br><font size=2 face="sans-serif">
comm =
newcomm;</font>
<br><font size=2 face="sans-serif">
}</font>
<br>
<br><font size=2 face="sans-serif">
MPI_Comm_size(comm, size);</font>
<br><font size=2 face="sans-serif">
</font>
<br><font size=2 face="sans-serif">
done = do_computation(buffer, size);</font>
<br>
<br><font size=2 face="sans-serif">
/* Let's agree for sure if we are done with
the computation */</font>
<br><font size=2 face="sans-serif">
</font>
<br><font size=2 face="sans-serif">
MPI_Barrier(comm); // don't check the
error code, this is just to "catch" invalidate messages</font>
<br><font size=2 face="sans-serif">
MPI_Comm_agreement(comm, &done);</font>
<br>
<br><font size=2 face="sans-serif"> while
(!done);</font>
<br>
<br><font size=2 face="sans-serif"> MPI_Finalize();</font>
<br>
<br><font size=2 face="CMR10">because a rank may enter the barrier, get
knocked out by the call to invalidate and then go on to call MPI_Comm_agreement
anyway. So we can try the following:</font>
<br>
<br><font size=2 face="sans-serif"> do
{</font>
<br>
<br><font size=2 face="sans-serif">
if (root) read_some_data_from_a_file(buffer);</font>
<br><font size=2 face="sans-serif">
</font>
<br><font size=2 face="sans-serif">
err = MPI_Bcast(buffer, .... ,root, comm);</font>
<br><font size=2 face="sans-serif">
if (err) {</font>
<br><font size=2 face="sans-serif">
MPI_Comm_invalidate(comm);</font>
<br><font size=2 face="sans-serif">
MPI_Comm_shrink(comm,
&newcomm);</font>
<br><font size=2 face="sans-serif">
MPI_Comm_free(&comm);</font>
<br><font size=2 face="sans-serif">
comm =
newcomm;</font>
<br><font size=2 face="sans-serif">
}</font>
<br>
<br><font size=2 face="sans-serif">
MPI_Comm_size(comm, size);</font>
<br><font size=2 face="sans-serif">
</font>
<br><font size=2 face="sans-serif">
done = do_computation(buffer, size);</font>
<br>
<br><font size=2 face="sans-serif">
/* Let's agree for sure if we are done with
the computation */</font>
<br><font size=2 face="sans-serif">
</font>
<br><font size=2 face="sans-serif">
err = MPI_Barrier(comm); </font>
<br><font size=2 face="sans-serif">
if (err) {</font>
<br><font size=2 face="sans-serif">
MPI_Comm_invalidate(comm);</font>
<br><font size=2 face="sans-serif">
MPI_Comm_shrink(comm,
&newcomm);</font>
<br><font size=2 face="sans-serif">
MPI_Comm_free(&comm);</font>
<br><font size=2 face="sans-serif">
comm =
newcomm;</font>
<br><font size=2 face="sans-serif">
}</font>
<br><font size=2 face="sans-serif">
MPI_Comm_agreement(comm, &done);</font>
<br>
<br><font size=2 face="sans-serif"> while
(!done);</font>
<br>
<br><font size=2 face="sans-serif"> MPI_Finalize();</font>
<br>
<br><font size=2 face="CMR10">But now we have done nothing more than move
the problem down a few lines. Some ranks may succeed the MPI_Barrier
and go on to MPI_Comm_agreement while others attempt to invalidate/shrink.
Is there a solution to this problem? How can one safely use
MPI_Comm_agreement and MPI_Comm_shrink in the same application?</font>
<br>
<br><font size=2 face="CMR10">Thanks,<br>
Dave </font>