<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:굴림;
panose-1:2 11 6 0 0 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"맑은 고딕";
panose-1:2 11 5 3 2 0 0 2 0 4;}
@font-face
{font-family:"\@맑은 고딕";}
@font-face
{font-family:"\@굴림";
panose-1:2 11 6 0 0 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:굴림;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:3.0cm 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="KO" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">I think an example is when you want to invoke a GPU kernel that depends on the arriving data.<o:p></o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Currently the host code has to wait until the MPI_Recv is complete before it is able to invoke the kernel, which can be avoided
if CUDA streams are supported in MPI.<o:p></o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph;text-autospace:none;word-break:break-hangul">
<span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Jaemin Choi<o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph;text-autospace:none;word-break:break-hangul">
<span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">PhD Candidate in Computer Science<o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph;text-autospace:none;word-break:break-hangul">
<span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Research Assistant at the Parallel Programming Laboratory<o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph;text-autospace:none;word-break:break-hangul">
<span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">University of Illinois Urbana-Champaign<o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="word-break:break-hangul"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Zhang, Junchao <jczhang@mcs.anl.gov>
<br>
<b>Sent:</b> Tuesday, February 9, 2021 9:09 PM<br>
<b>To:</b> Hybrid working group mailing list <mpiwg-hybridpm@lists.mpi-forum.org><br>
<b>Cc:</b> Jim Dinan <james.dinan@gmail.com>; Choi, Jaemin <jchoi157@illinois.edu><br>
<b>Subject:</b> Re: [mpiwg-hybridpm] Call for Topics<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">Is the host-side synchronization after communication needed? I thought no.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">--Junchao Zhang<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
<br>
<o:p></o:p></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span lang="EN-US">On Feb 9, 2021, at 8:19 PM, Choi, Jaemin via mpiwg-hybridpm <<a href="mailto:mpiwg-hybridpm@lists.mpi-forum.org">mpiwg-hybridpm@lists.mpi-forum.org</a>> wrote:<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks Jim, I must have missed the discussions from the last couple of meetings.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">We’ve recently implemented GPU-aware communication in Charm++ and Adaptive MPI using UCX, and just started to look into how we can avoid the host-side synchronizations
before/after communication.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">UCX supporting CUDA streams could be a solution, but I also wanted to explore other options as well. I wasn’t aware of libmp, I will definitely have a look at
that and NCCL to see how streams are integrated there.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Best,</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Jaemin Choi</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">PhD Candidate in Computer Science</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Research Assistant at the Parallel Programming Laboratory</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">University of Illinois Urbana-Champaign</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span class="apple-converted-space"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Jim
Dinan <<a href="mailto:james.dinan@gmail.com">james.dinan@gmail.com</a>><span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Tuesday, February 9, 2021 6:08 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Choi, Jaemin <<a href="mailto:jchoi157@illinois.edu">jchoi157@illinois.edu</a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span>Hybrid working group mailing list <<a href="mailto:mpiwg-hybridpm@lists.mpi-forum.org">mpiwg-hybridpm@lists.mpi-forum.org</a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [mpiwg-hybridpm] Call for Topics</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">We have been discussing this topic in the working group, and would be glad to have your input. If you're looking for something you can use today, NCCL is a good option. There's also a library called libmp that uses GPUDirect
Async for stream-based communication [1].<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> ~Jim.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">[1] <a href="https://urldefense.com/v3/__https:/github.com/gpudirect/libmp__;!!DZ3fjg!q1o2QIbKjMPaBeDqe6PrUc_204yx9iZ6vbdOXkEGLP--fM2nnq98nYn1-18ZkCEVqcUAjg$">https://github.com/gpudirect/libmp</a><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, Feb 9, 2021 at 3:09 PM Choi, Jaemin <<a href="mailto:jchoi157@illinois.edu">jchoi157@illinois.edu</a>> wrote:<o:p></o:p></span></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi Jim,</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Do you happen to know if there has been progress on supporting CUDA streams in the MPI standard?</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">I’d be interested to hear about this or other frameworks (e.g. NCCL) that support asynchronous communication via streams.</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Best,</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Jaemin Choi</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">PhD Candidate in Computer Science</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Research Assistant at the Parallel Programming Laboratory</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">University of Illinois Urbana-Champaign</span><span lang="EN-US"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><o:p></o:p></span></p>
</div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span class="apple-converted-space"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">mpiwg-hybridpm
<<a href="mailto:mpiwg-hybridpm-bounces@lists.mpi-forum.org" target="_blank">mpiwg-hybridpm-bounces@lists.mpi-forum.org</a>><span class="apple-converted-space"> </span><b>On Behalf Of<span class="apple-converted-space"> </span></b>Jim Dinan via mpiwg-hybridpm<br>
<b>Sent:</b><span class="apple-converted-space"> </span>Tuesday, February 9, 2021 1:10 PM<br>
<b>To:</b><span class="apple-converted-space"> </span>Hybrid working group mailing list <<a href="mailto:mpiwg-hybridpm@lists.mpi-forum.org" target="_blank">mpiwg-hybridpm@lists.mpi-forum.org</a>><br>
<b>Cc:</b><span class="apple-converted-space"> </span>Jim Dinan <<a href="mailto:james.dinan@gmail.com" target="_blank">james.dinan@gmail.com</a>><br>
<b>Subject:</b><span class="apple-converted-space"> </span>[mpiwg-hybridpm] Call for Topics</span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi All,<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">The Hybrid & Accelerator WG will meet tomorrow. The agenda so far is empty. Please let me know if you have any topics that you would like to discuss.<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Cheers,<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> ~Jim.<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:9.0pt;font-family:"Helvetica",sans-serif">_______________________________________________<br>
mpiwg-hybridpm mailing list<br>
<a href="mailto:mpiwg-hybridpm@lists.mpi-forum.org">mpiwg-hybridpm@lists.mpi-forum.org</a><br>
<a href="https://urldefense.com/v3/__https:/lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm__;!!DZ3fjg!v8lgZUpa2hzXDoU-brjVg-at7mu1xO718GusJEmp7kN4j0zcvSqy-mW8vCRRJgPGFi5huw$">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm</a></span><span lang="EN-US"><o:p></o:p></span></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>