<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Hi Edgar, My reading of this statement is that while we can provide mpi_assert_memory_alloc_kinds as an info object to MPI_ Comm_dup_with_info() to restrict the memory-alloc-kinds supported by the communicator, we cannot use mpi_assert_memory_alloc_kinds</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerx4ayh82" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerx4ayh82" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerx4ayh82" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerx4ayh82" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html>
<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerx4ayh82 { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonx4ayh82:hover, .pfptPrimaryButtonx4ayh82:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonx4ayh82:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Edgar,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<p class="elementToProof" style="margin-left: 0.5in;"><span style="font-family: "Arial", sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">My reading of this statement is that while we can provide
<i>mpi_assert_memory_alloc_kinds</i> as an info object to MPI_ Comm_dup_with_info() to restrict the memory-alloc-kinds supported by the communicator, we cannot use
<i>mpi_assert_memory_alloc_kinds</i> with MPI_Comm_set_info() (since this is not communicator creation), i.e. the key/value pair would be ignored. Consequently, there is no way for a user to restrict the memory_alloc_kinds used by a communicator created with
MPI_Comm_dup() (or any other constructor that does not take an info object as an argument). Is this interpretation correct?</span></p>
<p style="margin-left: 0.5in;"><span style="font-family: "Arial", sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> </span></p>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
IMHO, we should make assertions at object creation to make this tractable (not have to backtrack all the properties of the communicator or such), not after the fact.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
However, in as much as the standard is ambiguous, we have to choose.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Tony</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature" class="elementToProof">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Anthony Skjellum, PhD</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Professor of Computer Science</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Tennessee Technological University</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
email: askjellum@tntech.edu</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
cell: +1-205-807-4968</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> mpiwg-hybridpm <mpiwg-hybridpm-bounces@lists.mpi-forum.org> on behalf of Edgar Gabriel via mpiwg-hybridpm <mpiwg-hybridpm@lists.mpi-forum.org><br>
<b>Sent:</b> Wednesday, January 29, 2025 12:55 PM<br>
<b>To:</b> Hybrid working group mailing list <mpiwg-hybridpm@lists.mpi-forum.org><br>
<b>Cc:</b> Edgar Gabriel <edgar.gabriel1@outlook.com><br>
<b>Subject:</b> [mpiwg-hybridpm] mpi_memory_alloc_kinds questions/clarifications</font>
<div> </div>
</div>
<div>
<p align="center" style="text-align:center; background:white; margin:0px"><b><span style="font-size:12.0pt; color:red; background:white; font-family:"Calibri",sans-serif">External Email Warning</span></b></p>
<p align="center" style="text-align:center; background:white; margin:0px 12pt"><b><span style="font-size:12.0pt; color:red; font-family:"Calibri",sans-serif">This email originated from outside the university. Please use caution when opening attachments, clicking
links, or responding to requests.</span></b><span style="font-size:12.0pt"></span></p>
<hr>
<div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; height:0px; max-height:0px; opacity:0; overflow:hidden">
While working on the implementation of mpi_memory_alloc_kind info objects for Open MPI, I came across a couple of items that I wanted to clarify and/or have confirmed that my reading is correct. I understand that mpi_memory_alloc_kinds in the</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerStart</div>
<div dir="ltr" id="x_pfptBannerbn7h9ij" style="display:block!important; text-align:left!important; margin:16px 0px 16px 0px!important; padding:8px 16px 8px 16px!important; border-radius:4px!important; min-width:200px!important; background-color:#D0D8DC!important; background-color:#D0D8DC; border-top:4px solid #90a4ae!important; border-top:4px solid #90a4ae">
<div id="x_pfptBannerbn7h9ij" style="float:left!important; display:block!important; margin:0px 0px 1px 0px!important; max-width:600px!important">
<div id="x_pfptBannerbn7h9ij" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-weight:bold!important; font-weight:bold; font-size:14px!important; line-height:18px!important; line-height:18px">
This Message Is From an External Sender </div>
<div id="x_pfptBannerbn7h9ij" style="display:block!important; visibility:visible!important; background-color:#D0D8DC!important; color:#000000!important; color:#000000; font-weight:normal; font-family:'Arial',sans-serif!important; font-family:'Arial',sans-serif; font-size:12px!important; line-height:18px!important; line-height:18px; margin-top:2px!important">
This message came from outside your organization. </div>
</div>
<div style="clear:both!important; display:block!important; visibility:hidden!important; line-height:0!important; font-size:0.01px!important; height:0px">
</div>
</div>
<div style="display:none!important; display:none; visibility:hidden; font-size:1px; color:#ffffff; line-height:1px; max-height:0px; opacity:0; overflow:hidden">
ZjQcmQRYFpfptBannerEnd</div>
<style>
<!--
#x_pfptBannerbn7h9ij
{display:block!important;
visibility:visible!important;
opacity:1!important;
background-color:#D0D8DC!important;
max-width:none!important;
max-height:none!important}
-->
</style>
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Aptos}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif}
p.x_MsoListParagraph, li.x_MsoListParagraph, div.x_MsoListParagraph
{margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Aptos",sans-serif}
span.x_EmailStyle17
{font-family:"Arial",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:11.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
ol
{margin-bottom:0in}
ul
{margin-bottom:0in}
-->
</style>
<div class="x_WordSection1">
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif">While working on the implementation of mpi_memory_alloc_kind info objects for Open MPI, I came across a couple of items that I wanted to clarify and/or have confirmed that my reading is correct.</span></p>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif"> </span></p>
<ol start="1" type="1" style="margin-top:0in">
<li class="x_MsoListParagraph" style="margin-left:0in"><span style="font-family:"Arial",sans-serif">I understand that
<i>mpi_memory_alloc_kinds</i> in the world model is passed as an argument to mpiexec, and can be retrieved e.g. with MPI_Comm_get_info() on MPI_COMM_WORLD (there is an example in the side document to demonstrate the sequence). In addition, according to section
12.4.3, “In the World Model, an info hint passed to an MPI startup mechanism requests support for memory allocation kinds for all objects derived from the World Model.” So my reading is that if you do
</span></li></ol>
<p class="x_MsoNormal" style="margin-left:.5in; text-indent:.5in"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoNormal" style="margin-left:.5in; text-indent:.5in"><span style="font-family:"Arial",sans-serif">MPI_Comm_dup (MPI_COMM_WORLD, &comm_dup);</span></p>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif"> the value of
<i>mpi_memory_alloc_kinds</i> on comm_dup and MPI_COMM_WORLD should be the same.</span></p>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-family:"Arial",sans-serif">The part that triggers my question is related to the following sentence: “When the user sets the <i>mpi_assert_memory_alloc_kinds</i> info key on the input info object
for communicator creation, {…}, window creation, or file creation the implementation may assume that the memory for all communication buffers …”</span></p>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-family:"Arial",sans-serif">My reading of this statement is that while we can provide
<i>mpi_assert_memory_alloc_kinds</i> as an info object to MPI_ Comm_dup_with_info() to restrict the memory-alloc-kinds supported by the communicator, we cannot use
<i>mpi_assert_memory_alloc_kinds</i> with MPI_Comm_set_info() (since this is not communicator creation), i.e. the key/value pair would be ignored. Consequently, there is no way for a user to restrict the memory_alloc_kinds used by a communicator created with
MPI_Comm_dup() (or any other constructor that does not take an info object as an argument). Is this interpretation correct?</span></p>
<p class="x_MsoNormal" style="margin-left:.5in"><span style="font-family:"Arial",sans-serif"> </span></p>
<ol start="2" type="1" style="margin-top:0in">
<li class="x_MsoListParagraph" style="margin-left:0in"><span style="font-family:"Arial",sans-serif">There is some ambiguity in using the memkind info objects with MPI_Comm_spawn() and friends. The question really is whether it impacts the MPI_COMM_WORLD of
the spawned processes and/or also the resulting inter-communicator. Part of what is causing the confusion is that in section 12.8.2 its spelled out that the info-object passed to MPI_Comm_spawn() are “set of key-value pairs telling the runtime system where
and how to start the processes (handle, significant only at root)”.</span></li></ol>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif"> </span></p>
<ol start="3" type="1" style="margin-top:0in">
<li class="x_MsoListParagraph" style="margin-left:0in"><span style="font-family:"Arial",sans-serif">Finally, two comments to the side-document examples. The first one is a minor nitpick on using strcasecmp() when checking for a particular value. According to
section 11, “Both key and value are case sensitive”, so in theory strcmp() should suffice.</span></li></ol>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif">More relevant however is potentially something else. When user requests e.g. with
</span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> mpiexec -mpi-memory-alloc-kinds system,mpi,rocm:device -np 32 …</span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif">the MPI library is allowed to return more memory types than requested by the user, e.g. it would be valid for mpi_memory_alloc_kinds info to contain</span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> mpi,system,rocm:device,rocm:host,rocm:managed</span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif">or</span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> mpi,system,rocm</span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif">which is equivalent in my understanding, since device,host,and managed are all the memory types supported by the rocm memkind, and supporting all three of them is equivalent to not providing
any restrictors. Our examples in the side document could not handle that however, what we would need to do e.g. for the rocm testcase would be something like</span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> if (value is “rocm” (without restrictors)) || (value is “rocm:device”) …</span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoListParagraph"><span style="font-family:"Arial",sans-serif">(Not sure whether the same question arises for the default memkinds, e.g. “system,mpi” vs. “system,mpi:alloc_mem,mpi:win_allocate,mpi:win_allocate_shared”. Could a library have additional
restrictors for the mpi memory kind, in which case listing these three restrictors might not be equivalent to not listing any restrictors?)</span></p>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif"> </span></p>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif">Maybe we can discuss some of this in one of the upcoming meetings.</span></p>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif">Thanks</span></p>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif">Edgar</span></p>
<p class="x_MsoNormal"><span style="font-family:"Arial",sans-serif"> </span></p>
</div>
</div>
</div>
</body>
</html>