<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Akhil,<div class=""><br class=""></div><div class="">Thank you for your suggestion. This is an interesting area of API design for MPI. Let me jot down some notes in response to your points.</div><div class=""><br class=""></div><div class="">The MPI_Start function is used by both our proposed persistent collective communications and the existing persistent point-to-point communications. For consistency in the MPI Standard, any change to MPI_Start must be applied to point-to-point as well.</div><div class=""><br class=""></div><div class="">Our implementation work for persistent collective communication currently leverages point-to-point communication in a similar manner to your description of the tree broadcast. However, this is not required by the MPI Standard and is known to be a sub-optimal implementation choice. The interface design should not be determined by the needs of a poor implementation method.</div><div class=""><br class=""></div><div class="">All schedules for persistent collective communication operations involve multiple “rounds”. Each round concludes with a dependency on one or more remote MPI processes, i.e. a “wait”. This is not the case with point-to-point, where lower latency can be achieved with a fire-and-forget approach in some situations (ready mode or small eager protocol messages). Even for small buffer sizes, there is no ready mode or eager protocol for collective communications.</div><div class=""><br class=""></div><div class="">There is ongoing debate about the best method for implementing “wait”, e.g. active polling (spin wait) or signals (idle wait), etc. For collective operations, the inter-round “wait” could be avoided in many cases by using triggered operations - an incoming network packet is processed by the network hardware and triggers one or more response packets. Your “wait for receive, send to children” steps would then be “trigger store-and-foward on receive” programmed into the NIC itself. Having the CPU blocked would be a waste of resources for this implementation. This strongly argues that nonblocking should exist in the API, even if blocking is also added. Nonblocking already exists - MPI_Start.</div><div class=""><br class=""></div><div class="">With regards to interface naming, I would suggest MPI_Start_and_wait, and MPI_Start_and_test. You would also need to consider MPI_Startall_and_waitall and MPI_Startall_and_testall. I would avoid adding additional variants based on MPI_[Wait|Test][any|some].</div><div class=""><br class=""></div><div class="">There has been a lengthy debate about whether the persistent collective initialisation functions could/should be blocking or nonblocking. This issue is similar. One could envisage:</div><div class=""><br class=""></div><div class="">// fully non-blocking route - maximum opportunity for overlap - assumes normally slow network</div><div class="">MPI_Ireduce_init // begin optimisation of a reduction</div><div class="">MPI_Test // repeatedly test for completion of the optimisation of the reduction</div><div class=""><loop begin></div><div class="">MPI_Istart // begin the reduction communication</div><div class="">MPI_Test // repeatedly test for completion of the reduction communication</div><div class=""><loop end></div><div class="">MPI_Request_free // recover resources</div><div class=""><br class=""></div><div class=""><div class="">// fully blocking route - minimum opportunity for overlap - assumes infinitely fast network</div><div class="">MPI_Reduce_init // optimise a reduction, blocking</div><div class=""><loop begin></div><div class="">MPI_Start // do the reduction communication, blocking</div><div class=""><loop end></div><div class="">MPI_Request_free // recover resources</div></div><div class=""><br class=""></div><div class="">Some proposed optimisations take a long time and require collective communication, so we have chosen nonblocking initialisation. The current persistent communication workflow is initialise -> (start -> complete)* -> free, so we are not proposing to have the first MPI_Test in the example above. The existing MPI_Start is nonblocking so our proposal is basically the first of the examples above. It is a minimum change to the MPI Standard to achieve our main goal, i.e. permit a planning step for collective communications. It does not exclude or prevent additional proposals that extend the API in the manner that you suggest. However, such an extension would need a strong justification to succeed.</div><div class=""><br class=""></div><div class="">Cheers,</div><div class="">Dan.</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 16 May 2017, at 22:33, Langer, Akhil <<a href="mailto:akhil.langer@intel.com" class="">akhil.langer@intel.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;" class="">
<div class="">Hello, </div>
<div class=""><br class="">
</div>
<div class="">I want to propose an extension to persistent API to allow a blocking MPI_Start call. Currently, MPI_Start calls are non-blocking. So, proposal is something like MPI_Start (for blocking) and MPI_Istart (for non-blocking). Of course, to maintain backward
compatibility we may have to think of an alternative API. I am not proposing the exact API here. </div>
<div class=""><br class="">
</div>
<div class="">The motivation behind the proposal is that having the knowledge whether the corresponding MPI call is blocking or not can give better performance. For example, MPI_Isend followed by MPI_Wait is slower than the MPI_Send because internally MPI_Isend->MPI_Wait
has to allocate additional data structures (for example, request pointer) and do more work. Similarly, lets look at an example of a bcast collective operation. </div>
<div class=""><br class="">
</div>
<div class="">Tree based broadcast can be implemented in two ways:</div>
<ol class="">
<li class="">MPI_Recv (recv data from parent) -> FOREACHCHILD – MPI_Send (send data to children)</li><li class="">MPI_Irecv (recv data from parent) -> MPI_Wait(wait for recv to complete) -> FOREACHCHILD – MPI_Isend (send data to childrent) -> MPI_WaitAll (wait for sends to complete)</li></ol>
<div class="">Having only a non-blocking MPI_Start call forces only implementation 2 as implementation 1 has blocking MPI calls. However, implementation 1 can be significantly faster that implementation 2 for small message sizes.</div>
<div class=""><br class="">
</div>
<div class="">Looking forward to hear your feedback.</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Akhil </div>
<div class=""><br class="">
</div>
<div class="">
<div id="MAC_OUTLOOK_SIGNATURE" class=""></div>
</div>
</div>
</div></blockquote></div><br class=""></div></body></html>