<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Is the host-side synchronization after communication needed? I thought no.</div>
<div class=""><br class="">
</div>
<br class="">
<div class="">
<div>Thanks</div>
<div>--Junchao Zhang</div>
<div class=""><br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Feb 9, 2021, at 8:19 PM, Choi, Jaemin via mpiwg-hybridpm <<a href="mailto:mpiwg-hybridpm@lists.mpi-forum.org" class="">mpiwg-hybridpm@lists.mpi-forum.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks Jim, I must have missed the discussions from the last couple of meetings.<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">We’ve recently implemented GPU-aware communication in Charm++ and Adaptive MPI using UCX, and just started
to look into how we can avoid the host-side synchronizations before/after communication.<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">UCX supporting CUDA streams could be a solution, but I also wanted to explore other options as well.
I wasn’t aware of libmp, I will definitely have a look at that and NCCL to see how streams are integrated there.<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Best,<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림; text-align: justify;" class="">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Jaemin Choi<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림; text-align: justify;" class="">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">PhD Candidate in Computer Science<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림; text-align: justify;" class="">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Research Assistant at the Parallel Programming Laboratory<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림; text-align: justify;" class="">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">University of Illinois Urbana-Champaign<o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0cm 0cm;" class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><b class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="Apple-converted-space"> </span>Jim
Dinan <<a href="mailto:james.dinan@gmail.com" class="">james.dinan@gmail.com</a>><span class="Apple-converted-space"> </span><br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Tuesday, February 9, 2021 6:08 PM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>Choi, Jaemin <<a href="mailto:jchoi157@illinois.edu" class="">jchoi157@illinois.edu</a>><br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span>Hybrid working group mailing list <<a href="mailto:mpiwg-hybridpm@lists.mpi-forum.org" class="">mpiwg-hybridpm@lists.mpi-forum.org</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [mpiwg-hybridpm] Call for Topics<o:p class=""></o:p></span></div>
</div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div>
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class="">We have been discussing this topic in the working group, and would be glad to have your input. If you're looking for something you can use today, NCCL is a good
option. There's also a library called libmp that uses GPUDirect Async for stream-based communication [1].<o:p class=""></o:p></span></div>
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class=""> ~Jim.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class="">[1] <a href="https://urldefense.com/v3/__https:/github.com/gpudirect/libmp__;!!DZ3fjg!q1o2QIbKjMPaBeDqe6PrUc_204yx9iZ6vbdOXkEGLP--fM2nnq98nYn1-18ZkCEVqcUAjg$" style="color: blue; text-decoration: underline;" class="">https://github.com/gpudirect/libmp</a><o:p class=""></o:p></span></div>
</div>
</div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class=""><o:p class=""> </o:p></span></div>
<div class="">
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class="">On Tue, Feb 9, 2021 at 3:09 PM Choi, Jaemin <<a href="mailto:jchoi157@illinois.edu" style="color: blue; text-decoration: underline;" class="">jchoi157@illinois.edu</a>>
wrote:<o:p class=""></o:p></span></div>
</div>
<blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin-left: 4.8pt; margin-right: 0cm;" class="">
<div class="">
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hi Jim,</span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> </span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Do you happen to know if there has been progress on supporting CUDA streams in the MPI standard?</span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">I’d be interested to hear about this or other frameworks (e.g. NCCL) that support asynchronous communication
via streams.</span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> </span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Best,</span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> </span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림; text-align: justify;" class="">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Jaemin Choi</span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림; text-align: justify;" class="">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">PhD Candidate in Computer Science</span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림; text-align: justify;" class="">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">Research Assistant at the Parallel Programming Laboratory</span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림; text-align: justify;" class="">
<span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">University of Illinois Urbana-Champaign</span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""> </span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
<div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0cm 0cm;" class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><b class=""><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class="">From:</span></b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span class="Apple-converted-space"> </span>mpiwg-hybridpm
<<a href="mailto:mpiwg-hybridpm-bounces@lists.mpi-forum.org" target="_blank" style="color: blue; text-decoration: underline;" class="">mpiwg-hybridpm-bounces@lists.mpi-forum.org</a>><span class="Apple-converted-space"> </span><b class="">On Behalf Of<span class="Apple-converted-space"> </span></b>Jim
Dinan via mpiwg-hybridpm<br class="">
<b class="">Sent:</b><span class="Apple-converted-space"> </span>Tuesday, February 9, 2021 1:10 PM<br class="">
<b class="">To:</b><span class="Apple-converted-space"> </span>Hybrid working group mailing list <<a href="mailto:mpiwg-hybridpm@lists.mpi-forum.org" target="_blank" style="color: blue; text-decoration: underline;" class="">mpiwg-hybridpm@lists.mpi-forum.org</a>><br class="">
<b class="">Cc:</b><span class="Apple-converted-space"> </span>Jim Dinan <<a href="mailto:james.dinan@gmail.com" target="_blank" style="color: blue; text-decoration: underline;" class="">james.dinan@gmail.com</a>><br class="">
<b class="">Subject:</b><span class="Apple-converted-space"> </span>[mpiwg-hybridpm] Call for Topics</span><span lang="EN-US" class=""><o:p class=""></o:p></span></div>
</div>
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class=""> <o:p class=""></o:p></span></div>
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class="">Hi All,<o:p class=""></o:p></span></div>
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class=""> <o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class="">The Hybrid & Accelerator WG will meet tomorrow. The agenda so far is empty. Please let me know if you have any topics that you would like to discuss.<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class=""> <o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class="">Cheers,<o:p class=""></o:p></span></div>
</div>
<div class="">
<div style="margin: 0cm; font-size: 12pt; font-family: 굴림;" class=""><span lang="EN-US" class=""> ~Jim.<o:p class=""></o:p></span></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">mpiwg-hybridpm
mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="mailto:mpiwg-hybridpm@lists.mpi-forum.org" class="">mpiwg-hybridpm@lists.mpi-forum.org</a></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class=""><a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm</a></span></div>
</blockquote>
</div>
<br class="">
</body>
</html>