<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof"><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; font-weight: 400; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Guillaume
and I have been conversing off list about this issue today.</span></div>
<div class="elementToProof"><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; font-weight: 400; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);"><br>
</span></div>
<div class="elementToProof"><span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; font-weight: 400; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">There's
ongoing discussion about this topic in the Topologies WG.</span></div>
<div id="appendonsend"></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Jeff Squyres (jsquyres) <jsquyres@cisco.com><br>
<b>Sent:</b> Monday, November 27, 2023 1:58 PM<br>
<b>To:</b> Guillaume Mercier <guillaume.mercier@u-bordeaux.fr><br>
<b>Cc:</b> mpi-forum@lists.mpi-forum.org <mpi-forum@lists.mpi-forum.org><br>
<b>Subject:</b> Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info()</span>
<div> </div>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Ping; still waiting for some answers here.</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
My issue is that the text in MPI-4.1 is so broad as to be meaningless. I.e., both implementors and application developers can interpret it any way they want, and therefore the level and type of information delivered by different implementations could (will)
be wildly different in both their intent and execution. "Flexible" is one thing, but "so ambiguous as to be meaningless" is another.</div>
<div id="x_appendonsend"></div>
<hr style="direction: ltr; display: inline-block; width: 98%;">
<div id="x_divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Jeff Squyres (jsquyres) <jsquyres@cisco.com><br>
<b>Sent:</b> Thursday, November 16, 2023 11:50 AM<br>
<b>To:</b> Guillaume Mercier <guillaume.mercier@u-bordeaux.fr>; Jeff Squyres (jsquyres) <jsquyres@cisco.com><br>
<b>Cc:</b> mpi-forum@lists.mpi-forum.org <mpi-forum@lists.mpi-forum.org><br>
<b>Subject:</b> Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info()</span>
<div> </div>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Amusingly, my email client didn't show me any of your inline replies -- it <b>only</b> showed me your top-post reply. I didn't see your inline replies until after I sent my mail repeating/summarizing my original questions. Thanks, Outlook! 🙁</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Ok, let me reply to your inline comments:</div>
<ul data-editing-info="{"orderedStyleType":1,"unorderedStyleType":1}" style="direction: ltr; list-style-type: disc;">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; direction: ltr; color: rgb(0, 0, 0);">
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">If we're not supposed to list software or virtual instances, that's... problematic. Example: the Cisco NIC exports
a whole bunch of PCI virtual functions that are discovered by Linux, but it's really one PCI or Mezzanine card with tons of resources on it (each setting of resources is created dynamically and addressed separately). This gets very grey very fast.</span></li><ul style="direction: ltr; list-style-type: circle;">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">My overall point: the lines between hardware, firmware, and software are very, very
fuzzy these days. They're not likely to get any clearer, either.</span></div>
</li></ul>
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">For my question of asking what the value of the "true" and "false" values are: it's
not that <b>users</b> may request that resource allocations may change, it's that the MPI implementation, OS, drive, firmware, or even hardware may choose to change resource allocations. HPC/MPI applications tend to do this far less than non-HPC applications,
but it still does happen.</span></div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">You're calling it "flexible" -- but if different MPI implementations and/or different
vendors do things very differently, then what is the value of this mechanism for the application?</span></div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">Per your comment about the provider part of the URI intending to show where the information
came from, are you saying that the same resources can/should potentially be listed multiple times? E.g., openmpi://usnic-blah, cisco://usnic-blah, usnic://blah, pmix://nic/cisco/usnic/blah, hwloc://pci/235234/23424214, linux://net/usnic/32452345, ... ?</span></div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="direction: ltr;"><span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">I don't think you replied to some of my questions:</span></div>
</li><ul style="direction: ltr; list-style-type: circle;">
<li style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; font-weight: 400; direction: ltr; color: rgb(0, 0, 0);">
<span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; font-weight: 400; color: rgb(0, 0, 0);">What is the precise distinction between the "true" and "false" values
of the info keys?</span></li><li style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; font-weight: 400; direction: ltr; color: rgb(0, 0, 0);">
<span style="letter-spacing: normal; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; font-weight: 400; color: rgb(0, 0, 0);">What is the precise definition of when an implementation is required to
provide the same info keys/values between processes?</span></li></ul>
</ul>
<hr style="direction: ltr; display: inline-block; width: 98%;">
<div id="x_x_divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> mpi-forum <mpi-forum-bounces@lists.mpi-forum.org> on behalf of Jeff Squyres (jsquyres) via mpi-forum <mpi-forum@lists.mpi-forum.org><br>
<b>Sent:</b> Thursday, November 16, 2023 11:35 AM<br>
<b>To:</b> Guillaume Mercier <guillaume.mercier@u-bordeaux.fr><br>
<b>Cc:</b> mpi-forum@lists.mpi-forum.org <mpi-forum@lists.mpi-forum.org><br>
<b>Subject:</b> Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info()</span>
<div> </div>
</div>
<div style="direction: ltr; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Ok, that's a fine intent. But all my specific questions remain -- e.g.,</div>
<ul data-editing-info="{"orderedStyleType":1,"unorderedStyleType":1}" style="direction: ltr; list-style-type: disc;">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; direction: ltr; color: rgb(0, 0, 0);">
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">What is the precise distinction between the "true" and "false" values of the info keys?</span></li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; direction: ltr; color: rgb(0, 0, 0);">
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">What is the technical benefit of providing the "true" and "value" values to the user/application in the info keys</span></li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; direction: ltr; color: rgb(0, 0, 0);">
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">What items can be listed in these hardware info keys? (e.g., what about virtual or software-only devices)</span></li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; direction: ltr; color: rgb(0, 0, 0);">
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">What is the relationship between the software models listed as examples for the URI prefixes and the hardware that
they are supposed to represent?</span></li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; direction: ltr; color: rgb(0, 0, 0);">
<span style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">What is the precise definition of when an implementation is required to provide the same info keys/values between
processes?</span></li></ul>
<hr style="direction: ltr; display: inline-block; width: 98%;">
<div style="direction: ltr;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><b>From:</b> Guillaume Mercier<br>
<b>Sent:</b> Thursday, November 16, 2023 3:02 AM<br>
<b>To:</b> Jeff Squyres (jsquyres)<br>
<b>Cc:</b> mpi-forum@lists.mpi-forum.org<br>
<b>Subject:</b> Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info()</span></div>
<div style="direction: ltr;"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div style="direction: ltr;"><span style="font-size: 11pt;">Hi Jeff,<br>
<br>
Let me revise my first answer and be more specific on a couple<br>
of points you raise in your message.<br>
<br>
Remember that until MPI 4.1, there was no standard way to<br>
provide a value to the "mpi_hw_resource_type" info key that can guide<br>
the splitting of communicators on hardware-basis<br>
(i.e. with a call to MPI_comm_split_type with MPI_COMM_TYPE_HW_GUIDED<br>
as the input split_type value). MPI_Get_hw_resource_info fills<br>
this gap and makes applications that rely on this mechanism more<br>
portable than previously.<br>
<br>
On 16/11/2023 03:09, Jeff Squyres (jsquyres) via mpi-forum wrote:<br>
<br>
> 2. For example, my company makes a piece of hardware that can have<br>
> thousands of virtual NICs on it, and those virtual NICs might<br>
> even migrate around to different pieces of hardware (e.g., they<br>
> can migrate between different fiber optic outputs on the same<br>
> NIC). MPI processes are assigned to a virtual NIC, not a<br>
> hardware NIC. Am I allowed to include a reference to these<br>
> virtual NICs in the keys/values that are returned (since the<br>
> Linux device name refers to a virtual entity, not necessarily a<br>
> specific set of hardware)? If so, how do I determine the<br>
> true/false value to assign?<br>
<br>
On second thoughts, since these virtual NICs are software "instances"<br>
(for the lack of a better word), I'm not sure that they should be<br>
listed as keys in the resulting MPI_Info object. I'd like to discuss<br>
this more with you.<br>
<br>
> 3. The text states that the info keys/values are specific to the<br>
> point of time when the call is made. p446:11-12 even explicitly<br>
> states that the process and/or its hardware restrictions may<br>
> change over time. So even if I grokked what "restricted to a<br>
> single instance of a hardware resource of that type" is intended<br>
> to mean, if things can change -- and they can -- what is the<br>
> point of giving a true or false value to the user?<br>
<br>
Things can change, but not systematically. I don't think that current<br>
applications modify the binding of their MPI process that often.<br>
Therefore, in the majority of cases, the information you get after<br>
the first call to the procedure is likely to remain valid until<br>
the application's end.<br>
<br>
<br>
> 4.<br>
> Is the intent that keys will include a specific, unique<br>
> reference to an instance of "hardware" (e.g., a PCI address)?<br>
> If so, then the value of "true" and "false" becomes even more<br>
> nebulous (or meaningless). E.g., if I list a key containing<br>
> "cisco-nic-12bc83fde9" to indicate a specific NIC, what is the<br>
> exact "hardware resource of that type", and/or how would an<br>
> application know that "cisco-nic-12bc83fde9" and<br>
> "cisco-nic-bbbbbbbbb" are of the same "hardware resource type"?<br>
<br>
In this case, both guillaume://cisco-nic set to "true" AND<br>
jeffS://cisco-nic-12bc83fde9 set to "true" seem acceptable to me.<br>
It befalls the user to pick a provider and thus to consider which<br>
information should be effectively used.<br>
MPI_Get_hw_resource_info "only" fills the gap<br>
between the application and the lower-level mechanims that can be used<br>
to retrieve this kind of information without resorting to call this<br>
lower-level mechanims directly in the application.<br>
<br>
> 5. I can imagine that there could be many different scenarios here;<br>
> can someone provide some guidance on what exactly an<br>
> implementation is supposed to do here? This text seems to be...<br>
> ambiguous.<br>
<br>
What you call ambiguous, I would call flexible ;)<br>
But joking aside, the text can surely be improved and I'd more<br>
than happy to take your input into account and come up with an<br>
even better version for MPI 4.2 or 5.0.<br>
<br>
> 2. The AtoI in p445:42-46 says that we should use URIs with a type of<br>
> "openmpi://" or "hwloc://" or "pmix://" or "openmpi://" or<br>
> "slurm://" or ...<br>
> 1. All of these are software models (although hwloc's data refers<br>
> to either hardware or to software devices that correspond to<br>
> some form of hardware -- although that's not always clear, either).<br>
<br>
The provider only indicate where the information comes from, as two<br>
different sources might report slightly different things. I'll take your<br>
previous "cisco-nic "example: hwloc might choose to report only a<br>
"cisco-nic" type while Cisco's tool might report more precise<br>
information. I think it would be detrimental to the user to not report<br>
all possible informations. Then about software models, I would surely<br>
qualify "openmp" as a software model but not the others.<br>
<br>
<br>
> 2. The use of software models in the text is confusing, because the<br>
> routine has "hw" in its name, strongly implying that there's<br>
> supposed to be a direct tie-in to hardware.<br>
<br>
Software models are only used as potential providers, nothing more.<br>
Fundamentally I don't see the difference with what hwloc does<br>
(information reporting) and what this function does. Or maybe I didn't<br>
understand your comment right?<br>
<br>
Cheers,<br>
Guillaume</span></div>
</body>
</html>