<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
While working on the implementation of mpi_memory_alloc_kind info objects for Open MPI, I came across a couple of items that I wanted to clarify and/or have confirmed that my reading is correct. I understand that mpi_memory_alloc_kinds in the</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerbn7h9ij" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerbn7h9ij" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerbn7h9ij" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerbn7h9ij" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerbn7h9ij { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonbn7h9ij:hover, .pfptPrimaryButtonbn7h9ij:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonbn7h9ij:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Arial",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1980527206;
mso-list-type:hybrid;
mso-list-template-ids:-1205467422 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">While working on the implementation of mpi_memory_alloc_kind info objects for Open MPI, I came across a couple of items that I wanted to clarify and/or have confirmed that my reading is correct.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><span style="font-family:"Arial",sans-serif">I understand that
<i>mpi_memory_alloc_kinds</i> in the world model is passed as an argument to mpiexec, and can be retrieved e.g. with MPI_Comm_get_info() on MPI_COMM_WORLD (there is an example in the side document to demonstrate the sequence). In addition, according to section
12.4.3, “In the World Model, an info hint passed to an MPI startup mechanism requests support for memory allocation kinds for all objects derived from the World Model.” So my reading is that if you do
<o:p></o:p></span></li></ol>
<p class="MsoNormal" style="margin-left:.5in;text-indent:.5in"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:.5in"><span style="font-family:"Arial",sans-serif">MPI_Comm_dup (MPI_COMM_WORLD, &comm_dup);<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"> the value of
<i>mpi_memory_alloc_kinds</i> on comm_dup and MPI_COMM_WORLD should be the same.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Arial",sans-serif">The part that triggers my question is related to the following sentence: “When the user sets the <i>mpi_assert_memory_alloc_kinds</i> info key on the input info object
for communicator creation, {…}, window creation, or file creation the implementation may assume that the memory for all communication buffers …”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Arial",sans-serif">My reading of this statement is that while we can provide
<i>mpi_assert_memory_alloc_kinds</i> as an info object to MPI_ Comm_dup_with_info() to restrict the memory-alloc-kinds supported by the communicator, we cannot use
<i>mpi_assert_memory_alloc_kinds</i> with MPI_Comm_set_info() (since this is not communicator creation), i.e. the key/value pair would be ignored. Consequently, there is no way for a user to restrict the memory_alloc_kinds used by a communicator created with
MPI_Comm_dup() (or any other constructor that does not take an info object as an argument). Is this interpretation correct?<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<ol style="margin-top:0in" start="2" type="1">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><span style="font-family:"Arial",sans-serif">There is some ambiguity in using the memkind info objects with MPI_Comm_spawn() and friends. The question really is whether it impacts
the MPI_COMM_WORLD of the spawned processes and/or also the resulting inter-communicator. Part of what is causing the confusion is that in section 12.8.2 its spelled out that the info-object passed to MPI_Comm_spawn() are “set of key-value pairs telling the
runtime system where and how to start the processes (handle, significant only at root)”.<o:p></o:p></span></li></ol>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<ol style="margin-top:0in" start="3" type="1">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1"><span style="font-family:"Arial",sans-serif">Finally, two comments to the side-document examples. The first one is a minor nitpick on using strcasecmp() when checking for a particular
value. According to section 11, “Both key and value are case sensitive”, so in theory strcmp() should suffice.<o:p></o:p></span></li></ol>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif">More relevant however is potentially something else. When user requests e.g. with
<o:p></o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"> mpiexec -mpi-memory-alloc-kinds system,mpi,rocm:device -np 32 …<o:p></o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif">the MPI library is allowed to return more memory types than requested by the user, e.g. it would be valid for mpi_memory_alloc_kinds info to contain<o:p></o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"> mpi,system,rocm:device,rocm:host,rocm:managed<o:p></o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif">or<o:p></o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"> mpi,system,rocm<o:p></o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif">which is equivalent in my understanding, since device,host,and managed are all the memory types supported by the rocm memkind, and supporting all three of them is equivalent to not providing
any restrictors. Our examples in the side document could not handle that however, what we would need to do e.g. for the rocm testcase would be something like<o:p></o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"> if (value is “rocm” (without restrictors)) || (value is “rocm:device”) …<o:p></o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoListParagraph"><span style="font-family:"Arial",sans-serif">(Not sure whether the same question arises for the default memkinds, e.g. “system,mpi” vs. “system,mpi:alloc_mem,mpi:win_allocate,mpi:win_allocate_shared”. Could a library have additional
restrictors for the mpi memory kind, in which case listing these three restrictors might not be equivalent to not listing any restrictors?)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Maybe we can discuss some of this in one of the upcoming meetings.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Edgar<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
</div>
</body>
</html>