<div dir="ltr">You are right that I am assuming that if an NBC completes at one rank, I can reason about the state of that operation at another. I can certainly come up with scenarios where that is dangerous.<div><br></div><div>Jeff</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Aug 15, 2020 at 9:57 AM Richard Graham <<a href="mailto:richgraham@nvidia.com">richgraham@nvidia.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="auto">I think you start going down a slippery slope when you start to guess about the state of a resource that has been put under the control on another entity, without that entity explicitly declaring something.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Rich</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div id="gmail-m_-6905357919524417461composer_signature" dir="auto">
<div dir="auto" style="font-size:85%;color:rgb(87,87,87)">Sent via the Samsung Galaxy S9, an AT&T 5G Evolution capable smartphone</div>
</div>
<div dir="auto"><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>-------- Original message --------</div>
<div>From: Jeff Hammond via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank">mpi-forum@lists.mpi-forum.org</a>> </div>
<div>Date: 8/15/20 12:51 (GMT-05:00) </div>
<div>To: "Skjellum, Anthony" <<a href="mailto:Tony-Skjellum@utc.edu" target="_blank">Tony-Skjellum@utc.edu</a>> </div>
<div>Cc: Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>>, Main MPI Forum mailing list <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank">mpi-forum@lists.mpi-forum.org</a>>
</div>
<div>Subject: Re: [Mpi-forum] MPI_Request_free restrictions </div>
<div><br>
</div>
<div>
<table bgcolor="#FFEB9C" border="1">
<tbody>
<tr>
<td><font face="verdana" color="black" size="1"><b>External email: Use caution opening links or attachments</b>
</font></td>
</tr>
</tbody>
</table>
<br>
<div>
<div dir="ltr">
<div>Yes, but do we think the use case of never detecting completion is likely? I am not arguing for that, but rather that users might free the request but then detect completion another way, such as by waiting on the request on a subset of processes.</div>
<div><br>
</div>
<div>Jeff</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Aug 12, 2020 at 5:40 PM Skjellum, Anthony <<a href="mailto:Tony-Skjellum@utc.edu" target="_blank">Tony-Skjellum@utc.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
FYI, one argument (also used to force us to add restrictions on MPI persistent collective initialization to be blocking)... <span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">The MPI_Request_free on an NBC poses a
problem for the cases where there are array types</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
posed (e.g., Alltoallv/w)... It will not be knowable to the application if the vectors are in use by MPI still after </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
the free on an active request. We do *not* <span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">mandate that the MPI implementation copy such arrays currently, so they are effectively "held as unfreeable" by the MPI </span><span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">implementation
till MPI_Finalize. The user cannot deallocate them in a correct program till after MPI_Finalize. </span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">Another effect for NBC of releasing an active request, IMHO, is that you don't know when send buffers are free to be deallocated or receive buffers are free to be
deallocated... since you don't know when the transfer is complete OR the buffers are no longer used by MPI (till after MPI_Finalize).</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt">Tony</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-6905357919524417461gmail-m_4110230693676918912Signature">
<div>
<div id="gmail-m_-6905357919524417461gmail-m_4110230693676918912divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top:0px;margin-bottom:0px">Anthony Skjellum, PhD</p>
<p style="margin-top:0px;margin-bottom:0px">Professor of Computer Science and Chair of Excellence</p>
<p style="margin-top:0px;margin-bottom:0px">Director, SimCenter</p>
<p style="margin-top:0px;margin-bottom:0px">University of Tennessee at Chattanooga (UTC)</p>
<p style="margin-top:0px;margin-bottom:0px"><a href="mailto:tony-skjellum@utc.edu" target="_blank">tony-skjellum@utc.edu</a> [or
<a href="mailto:skjellum@gmail.com" target="_blank">skjellum@gmail.com</a>]</p>
<p style="margin-top:0px;margin-bottom:0px">cell: 205-807-4968</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
</div>
</div>
</div>
</div>
<div id="gmail-m_-6905357919524417461gmail-m_4110230693676918912appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-6905357919524417461gmail-m_4110230693676918912divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> mpi-forum <<a href="mailto:mpi-forum-bounces@lists.mpi-forum.org" target="_blank">mpi-forum-bounces@lists.mpi-forum.org</a>>
on behalf of Jeff Hammond via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank">mpi-forum@lists.mpi-forum.org</a>><br>
<b>Sent:</b> Saturday, August 8, 2020 12:07 PM<br>
<b>To:</b> Main MPI Forum mailing list <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank">mpi-forum@lists.mpi-forum.org</a>><br>
<b>Cc:</b> Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a>><br>
<b>Subject:</b> Re: [Mpi-forum] MPI_Request_free restrictions</font>
<div> </div>
</div>
<div dir="auto">We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. <br>
<br>
<div dir="ltr">Sent from my iPhone</div>
<div dir="ltr"><br>
<blockquote type="cite">On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank">mpi-forum@lists.mpi-forum.org</a>> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr"> Folks,
<div><br>
</div>
<div>Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not for Request_free.
AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would like to free NBC
requests so they don’t have to wait for the operation to complete in some situations.</div>
<div><br>
</div>
<div>Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think of now.</div>
<div><br>
</div>
<div>Can someone clarify or remind me what the reason was?</div>
<div><br>
</div>
<div>Regards,</div>
<div><br>
</div>
<div> — Pavan</div>
<div><br>
</div>
<div>MPI-3.1 standard, page 197, lines 26-27:</div>
<div><br>
</div>
<div>“<span style="font-size:11pt;font-family:CMR10">It is erroneous to call </span>
<span style="font-size:11pt;font-family:CMSS10">MPI</span><span style="font-size:11pt;font-family:CMTT10">_</span><span style="font-size:11pt;font-family:CMSS10">REQUEST</span><span style="font-size:11pt;font-family:CMTT10">_</span><span style="font-size:11pt;font-family:CMSS10">FREE
</span><span style="font-size:11pt;font-family:CMR10">or </span><span style="font-size:11pt;font-family:CMSS10">MPI</span><span style="font-size:11pt;font-family:CMTT10">_</span><span style="font-size:11pt;font-family:CMSS10">CANCEL
</span><span style="font-size:11pt;font-family:CMR10">for a request associated with a nonblocking collective operation.</span>”</div>
<div><br>
</div>
<span>_______________________________________________</span><br>
<span>mpi-forum mailing list</span><br>
<span><a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank">mpi-forum@lists.mpi-forum.org</a></span><br>
<span><a href="https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mpi-forum.org%2Fmailman%2Flistinfo%2Fmpi-forum&data=02%7C01%7Crichgraham%40nvidia.com%7Cda892ae3f6da411c982908d8413b8181%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637331071121370760&sdata=L1UxfurOJSMZu0VlEgq5tl7sqghnlzVx%2Fr%2BOxVq4tdQ%3D&reserved=0" target="_blank">https://lists.mpi-forum.org/mailman/listinfo/mpi-forum</a></span><br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Jeff Hammond<br>
<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br>
<a href="https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fjeffhammond.github.io%2F&data=02%7C01%7Crichgraham%40nvidia.com%7Cda892ae3f6da411c982908d8413b8181%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637331071121380752&sdata=UW5krprJGbRnlqB3YHXrt82QRM8fJ5gn3bo%2B63E1D6g%3D&reserved=0" target="_blank">http://jeffhammond.github.io/</a></div>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>