<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Quincey,
<div class=""><br class="">
</div>
<div class="">The MPI barrier operation (whether blocking, nonblocking, or persistent) does not guarantee “memory fence” semantics (either for the content of memory or the content of files).</div>
<div class=""><br class="">
</div>
<div class="">Perhaps you are looking for MPI_FILE_SYNC?</div>
<div class=""><br class="">
</div>
<div class="">"If other processes have made updates to the storage device, then all such updates become visible to subsequent reads of fh by the calling process.” §13.6.1</div>
<div class=""><br class="">
</div>
<div class="">"MPI_FILE_SYNC is a collective operation.” §13.6.1</div>
<div class=""><br class="">
</div>
<div class="">Used correctly (user must locally complete their I/O operations before calling it), this does provide a “fence”-like guarantee *for the file*, which is what your code looks like you are attempting. That is, all remote writes to the file that were
initiated remotely (and locally completed at the remote process) before the matching remote call to MPI_FILE_SYNC are guaranteed to be visible in the file using subsequent locally issued MPI read operations once the local call to MPI_FILE_SYNC completes locally.</div>
<div class=""><br class="">
</div>
<div class="">There is currently no nonblocking or persistent expression of this MPI procedure - watch this space: this is on the to-do list for MPI-Next.</div>
<div class=""><br class="">
</div>
<div class="">As Jim points out, the performance problem you note is most likely due to the implicit MPI_FILE_SYNC-like synchronisation done internally by MPI during the MPI_FILE_CLOSE procedure call. All enqueued file operations targeting the file will be
flushed to the file during MPI_FILE_CLOSE. If file operations are not flushed to the file concurrently with the application stuff or the MPI communication operations, then they will still be enqueued when MPI_FILE_CLOSE is called.<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="Apple-interchange-newline">
Cheers,</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Dan.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Architect (HPC Research)<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—</div>
</div>
</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 14 Aug 2020, at 17:32, Quincey Koziol via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Dan,
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>I believe that Pavan was referring to my conversation with him about MPI_Request_free. Here’s my situation: I’d like to use MPI_Ibarrier as a form of “memory fence” between some of the
metadata reads and writes in HDF5. Here’s some [very] simplified pseudocode for what I’d like to do:</div>
<div class=""><br class="">
</div>
<div class="">===============================</div>
<div class=""><br class="">
</div>
<div class=""><open HDF5 file> // sets up a communicator for internal HDF5 communication about this file</div>
<div class=""><br class="">
</div>
<div class="">do {</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>MPI_Ibarrier(<file’s communicator>, &request);</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span><application stuff></div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>// HDF5 operation:</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>if(<operation is read or write>) {</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>MPI_Wait(&request);</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span><perform read / write></div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>}</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>else { // operation is a file close</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>MPI_Request_free(&request);</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>MPI_File_close(…);</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>MPI_Comm_free(<file’s communicator>);</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>}</div>
<div class="">} while (<file is open>);</div>
<div class=""><br class="">
</div>
<div class="">===============================</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>What I am really trying to avoid is calling MPI_Wait at file close, since it is semantically unnecessary and only increases the latency from the application’s perspective. If I can’t
call MPI_Request_free on a nonblocking collective operation’s request (and it looks like I can’t, right now), I will have to put the request and file’s communicator into a “cleanup” list that is polled periodically [on each rank] with MPI_Test and disposed
of when the nonblocking barrier completes locally.</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>So, I’d really like to be able to call MPI_Request_free on the nonblocking barrier’s request.</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>Thoughts?</div>
<div class=""><br class="">
</div>
<div class=""><span class="Apple-tab-span" style="white-space:pre"></span>Quincey</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 13, 2020, at 9:07 AM, HOLMES Daniel via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Jim,
<div class=""><br class="">
</div>
<div class="">To be clear, I think that MPI_CANCEL is evil and should be removed from the MPI Standard entirely at the earliest convenience.</div>
<div class=""><br class="">
</div>
<div class="">I am certainly not arguing that it be permitted for more MPI operations.</div>
<div class=""><br class="">
</div>
<div class="">I thought the discussion was focused on MPI_REQUEST_FREE and whether or not it can/should be used on an active request.</div>
<div class=""><br class="">
</div>
<div class="">If a particular MPI implementation does not keep a reference to the request between MPI_RPUT and MPI_REQUEST_FREE, but needs that reference to process the completion event, then that MPI implementation would be required to keep a reference to
the request from MPI_REQUEST_FREE until that important task had been done, perhaps until the close epoch call. This requires no new memory because the user is giving up their reference to the request, so MPI can safely use the request it is passed in MPI_REQUEST_FREE
without copying it. As you say, MPI takes over the responsibility for processing the completion event.</div>
<div class=""><br class="">
</div>
<div class="">Your question about why the implementation should be required to take on this complexity is a good one. That, I guess, is why freeing any active request is a bad idea. MPI is required to differentiate completion of individual operations (so it
can implement MPI_WAIT) but that means something must process completion at some point for each individual operation. In RMA, that responsibility can be discharged earlier than in other parts of the MPI interface, but the real question is “why should MPI offer
to take on this responsibility in the first place?”</div>
<div class=""><br class="">
</div>
<div class="">Thanks, that helps (me at least).<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="Apple-interchange-newline">
Cheers,</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Dan.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Architect (HPC Research)<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—</div>
</div>
</div>
</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 13 Aug 2020, at 14:43, Jim Dinan <<a href="mailto:james.dinan@gmail.com" class="">james.dinan@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">The two cases you mentioned would have the same behavior at an application level. However, there may be important differences in the implementation of each operation. For example, an MPI_Put operation may be configured to not generate a
completion event, whereas an MPI_Rput would. The library may be relying on the user to make a call on the request to process the event and clean up resources. The implementation can take over this responsibility if the user cancels the request, but why should
we ask implementers to take on this complexity and overhead?
<div class=""><br class="">
</div>
<div class="">My $0.02 is that MPI_Cancel is subtle and complicated, and we should be very careful about where we allow it. I don't see the benefit to the programming model outweighing the complexity and overhead in the MPI runtime for the case of MPI_Rput.
I also don't know that we were careful enough in specifying the RMA memory model that a canceled request-based RMA operation will still have well-defined behavior. My understanding is that MPI_Cancel is required primarily for canceling receive requests to
meet MPI's quiescent shutdown requirement.
<div class=""><br class="">
</div>
<div class=""> ~Jim.</div>
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Aug 13, 2020 at 8:11 AM HOLMES Daniel via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" class="">Hi all,
<div class=""><br class="">
</div>
<div class="">To increase my own understanding of RMA, what is the difference (if any) between a request-based RMA operation where the request is freed without being completed and before the epoch is closed and a “normal” RMA operation?</div>
<div class=""><br class="">
</div>
<div class="">MPI_LOCK() ! or any other "open epoch at origin" procedure call</div>
<div class="">doUserWorkBefore()</div>
<div class="">MPI_RPUT(&req)</div>
<div class="">MPI_REQUEST_FREE(&req)</div>
<div class="">doUserWorkAfter()</div>
<div class="">MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call</div>
<div class=""><br class="">
</div>
<div class="">vs:</div>
<div class=""><br class="">
</div>
<div class="">MPI_LOCK() ! or any other "open epoch at origin" procedure call</div>
<div class="">doUserWorkBefore()</div>
<div class="">MPI_PUT()</div>
<div class="">
<div class="">doUserWorkAfter()</div>
<div class="">MPI_UNLOCK() ! or the matching “close epoch at origin" procedure call</div>
<div class=""><br class="">
</div>
<div class="">Is this a source-to-source translation that is always safe in either direction?</div>
<div class=""><br class="">
</div>
<div class="">In RMA, in contrast to the rest of MPI, there are two opportunities for MPI to “block” and do non-local work to complete an RMA operation: 1) during MPI_WAIT for the request (if any - the user may not be given a request or the user may choose
to free the request without calling MPI_WAIT or the user might call nonblocking MPI_TEST) and 2) during the close epoch procedure, which is always permitted to be sufficiently non-local to guarantee that the RMA operation is complete and its freeing stage
has been done. It seems that a request-based RMA operation becomes identical to a “normal” RMA operation if the user calls MPI_REQUEST_FREE on the request. This is like “freeing" the request from a nonblocking point-to-point operation but without the guarantee
of a later synchronisation procedure that can actually complete the operation and actually do the freeing stage of the operation.</div>
<div class=""><br class="">
</div>
<div class="">In collectives, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for collectives.</div>
<div class="">
<div class="">In point-to-point, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for point-to-point.</div>
</div>
<div class="">
<div class="">In file operations, there is no “ensure all operations so far are now done” procedure call because there is no concept of epoch for file operations. (There is MPI_FILE_SYNC but it is optional so MPI cannot rely on it being called.)</div>
</div>
<div class="">In these cases, the only non-local procedure that is guaranteed to happen is MPI_FINALIZE, hence all outstanding non-local work needed by the “freed” operation might be delayed until that procedure is called.</div>
<div class=""><br class="">
</div>
<div class="">The issue with copying parameters is also moot because all of them are passed-by-value (implicitly copied) or are data-buffers and covered by “conflicting accesses” RMA rules.</div>
<div class=""><br class="">
</div>
<div class="">Thus, to me it seems to me that RMA is a very special case - it could support different semantics, but that does not provide a good basis for claiming that the rest of the MPI Standard can support those different semantics - unless we introduce
an epoch concept into the rest of the MPI Standard. This is not unreasonable: the notifications in GASPI, for example, guarantee completion of not just the operation they are attached to but *all* operations issued in the “queue” they represent since the last
notification. Their queue concept serves the purpose of an epoch. I’m sure there are other examples in other APIs. It seems to me likely that the proposal for MPI_PSYNC for partitioned communication operations is moving in the direction of an epoch, although
limited to remote completion of all the partitions in a single operation, which accidentally guarantees that the operation can be freed locally using a local procedure.</div>
<div class=""><br class="">
</div>
<div class="">
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div dir="auto" style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;" class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
Cheers,</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
Dan.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Architect (HPC Research)<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" target="_blank" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
—</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">
—</div>
</div>
</div>
</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 13 Aug 2020, at 01:40, Skjellum, Anthony via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank" class="">mpi-forum@lists.mpi-forum.org</a>> wrote:</div>
<br class="">
<div class="">
<div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">
FYI, one argument (also used to force us to add restrictions on MPI persistent collective initialization to be blocking)... <span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">The MPI_Request_free on an NBC poses a problem for
the cases where there are array types</span></div>
<div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">
posed (e.g., Alltoallv/w)... It will not be knowable to the application if the vectors are in use by MPI still after </div>
<div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">
the free on an active request. We do *not* <span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">mandate that the MPI implementation copy such arrays currently, so they are effectively "held as unfreeable" by the MPI </span><span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">implementation
till MPI_Finalize. The user cannot deallocate them in a correct program till after MPI_Finalize. </span></div>
<div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class=""><br class="">
</span></div>
<div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">Another effect for NBC of releasing an active request, IMHO, is that you don't know when send buffers are free to be deallocated or receive buffers are free to be deallocated...
since you don't know when the transfer is complete OR the buffers are no longer used by MPI (till after MPI_Finalize).</span></div>
<div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class=""><br class="">
</span></div>
<div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">Tony</span></div>
<div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">
<br class="">
</div>
<div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">
<br class="">
</div>
<div style="font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">
<br class="">
</div>
<div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt" class="">
<br class="">
</div>
<div id="gmail-m_-8482721573127967263Signature" class="">
<div class="">
<div id="gmail-m_-8482721573127967263divtagdefaultwrapper" dir="ltr" style="font-size:12pt;font-family:Calibri,Helvetica,sans-serif" class="">
<div style="margin-top:0px;margin-bottom:0px" class="">Anthony Skjellum, PhD</div>
<div style="margin-top:0px;margin-bottom:0px" class="">Professor of Computer Science and Chair of Excellence</div>
<div style="margin-top:0px;margin-bottom:0px" class="">Director, SimCenter</div>
<div style="margin-top:0px;margin-bottom:0px" class="">University of Tennessee at Chattanooga (UTC)</div>
<div style="margin-top:0px;margin-bottom:0px" class=""><a href="mailto:tony-skjellum@utc.edu" target="_blank" class="">tony-skjellum@utc.edu</a><span class=""> </span> [or<span class=""> </span><a href="mailto:skjellum@gmail.com" target="_blank" class="">skjellum@gmail.com</a>]</div>
<div style="margin-top:0px;margin-bottom:0px" class="">cell: 205-807-4968</div>
<div style="margin-top:0px;margin-bottom:0px" class=""><br class="">
</div>
</div>
</div>
</div>
</div>
<div id="gmail-m_-8482721573127967263appendonsend" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">
</div>
<hr style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;display:inline-block;width:986.859px" class="">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline" class=""></span>
<div id="gmail-m_-8482721573127967263divRplyFwdMsg" dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">
<font face="Calibri, sans-serif" style="font-size:11pt" class=""><b class="">From:</b><span class=""> </span>mpi-forum <<a href="mailto:mpi-forum-bounces@lists.mpi-forum.org" target="_blank" class="">mpi-forum-bounces@lists.mpi-forum.org</a>> on behalf of Jeff
Hammond via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank" class="">mpi-forum@lists.mpi-forum.org</a>><br class="">
<b class="">Sent:</b><span class=""> </span>Saturday, August 8, 2020 12:07 PM<br class="">
<b class="">To:</b><span class=""> </span>Main MPI Forum mailing list <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank" class="">mpi-forum@lists.mpi-forum.org</a>><br class="">
<b class="">Cc:</b><span class=""> </span>Jeff Hammond <<a href="mailto:jeff.science@gmail.com" target="_blank" class="">jeff.science@gmail.com</a>><br class="">
<b class="">Subject:</b><span class=""> </span>Re: [Mpi-forum] MPI_Request_free restrictions</font>
<div class=""> </div>
</div>
<div dir="auto" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">
We should fix the RMA chapter with an erratum. I care less about NBC but share your ignorance of why it was done that way. <br class="">
<br class="">
<div dir="ltr" class="">Sent from my iPhone</div>
<div dir="ltr" class=""><br class="">
<blockquote type="cite" class="">On Aug 8, 2020, at 6:51 AM, Balaji, Pavan via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank" class="">mpi-forum@lists.mpi-forum.org</a>> wrote:<br class="">
<br class="">
</blockquote>
</div>
<blockquote type="cite" class="">
<div dir="ltr" class=""> Folks,
<div class=""><br class="">
</div>
<div class="">Does someone remember why we disallowed users from calling MPI_Request_free on nonblocking collective requests? I remember the reasoning for not allowing cancel (i.e., the operation might have completed on some processes, but not all), but not
for Request_free. AFAICT, allowing the users to free the request doesn’t make any difference to the MPI library. The MPI library would simply maintain its own refcount to the request and continue forward till the operation completes. One of our users would
like to free NBC requests so they don’t have to wait for the operation to complete in some situations.</div>
<div class=""><br class="">
</div>
<div class="">Unfortunately, when I added the Rput/Rget operations in the RMA chapter, I copy-pasted that text into RMA as well without thinking too hard about it. My bad! Either the RMA committee missed it too, or they thought of a reason that I can’t think
of now.</div>
<div class=""><br class="">
</div>
<div class="">Can someone clarify or remind me what the reason was?</div>
<div class=""><br class="">
</div>
<div class="">Regards,</div>
<div class=""><br class="">
</div>
<div class=""> — Pavan</div>
<div class=""><br class="">
</div>
<div class="">MPI-3.1 standard, page 197, lines 26-27:</div>
<div class=""><br class="">
</div>
<div class="">“<span style="font-size:11pt;font-family:CMR10" class="">It is erroneous to call<span class=""> </span></span><span style="font-size:11pt;font-family:CMSS10" class="">MPI</span><span style="font-size:11pt;font-family:CMTT10" class="">_</span><span style="font-size:11pt;font-family:CMSS10" class="">REQUEST</span><span style="font-size:11pt;font-family:CMTT10" class="">_</span><span style="font-size:11pt;font-family:CMSS10" class="">FREE<span class=""> </span></span><span style="font-size:11pt;font-family:CMR10" class="">or<span class=""> </span></span><span style="font-size:11pt;font-family:CMSS10" class="">MPI</span><span style="font-size:11pt;font-family:CMTT10" class="">_</span><span style="font-size:11pt;font-family:CMSS10" class="">CANCEL<span class=""> </span></span><span style="font-size:11pt;font-family:CMR10" class="">for
a request associated with a nonblocking collective operation.</span>”</div>
<div class=""><br class="">
</div>
<span class="">_______________________________________________</span><br class="">
<span class="">mpi-forum mailing list</span><br class="">
<span class=""><a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank" class="">mpi-forum@lists.mpi-forum.org</a></span><br class="">
<span class=""><a href="https://lists.mpi-forum.org/mailman/listinfo/mpi-forum" target="_blank" class="">https://lists.mpi-forum.org/mailman/listinfo/mpi-forum</a></span><br class="">
</div>
</blockquote>
</div>
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline" class="">_______________________________________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">
<span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline" class="">mpi-forum
mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">
<a href="mailto:mpi-forum@lists.mpi-forum.org" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" class="">mpi-forum@lists.mpi-forum.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpi-forum" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank" class="">https://lists.mpi-forum.org/mailman/listinfo/mpi-forum</a></div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
mpi-forum mailing list<br class="">
<a href="mailto:mpi-forum@lists.mpi-forum.org" target="_blank" class="">mpi-forum@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpi-forum" rel="noreferrer" target="_blank" class="">https://lists.mpi-forum.org/mailman/listinfo/mpi-forum</a><br class="">
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
mpi-forum mailing list<br class="">
<a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpi-forum" class="">https://lists.mpi-forum.org/mailman/listinfo/mpi-forum</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
mpi-forum mailing list<br class="">
<a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a><br class="">
https://lists.mpi-forum.org/mailman/listinfo/mpi-forum<br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>