<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, "EmojiFont", "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hi Kathryn,</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"></p>
<div>The following use cases are important for Arm Forge (debugger DDT and profiler MAP).
<br>
<br>
<b>Use case 1: Launch an MPI job under the control of a tool. </b><br>
<br>
<i>Input:</i> The user's regular command line to start the MPI job, for example "mpirun -n 256 ./wave_c". Why? To make it as friction-less as possible to use a tool.
<br>
<br>
NB: The tool does not want to parse or interpret the MPI starter arguments, as the tool has no knowledge of all possible MPI starter arguments.
<br>
<br>
<i>Requirements: </i><br>
<br>
<ul style="margin-bottom: 0px; margin-top: 0px;">
<li><b>Hold MPI processes</b> for the tool to attach to them and to release them when attached.<br>
Q: Where are processes held? In MPI_Init or _start? </li><li><b>Spawn tool daemon</b> on nodes where job is running.<br>
Q: Co-spawn or separate?<br>
Q: One per node or one per process?<br>
Q: Environment? Same general environment as MPI processes instead of restricted environment (e.g. $HOME not set or chroot).<br>
</li><li><b>Acquire the process table</b> (rank, hostname, pid and executable path) of spawned MPI processes.<br>
Q: Only global proctable or additionally local (per node) proctable? </li><li><span style="color: rgb(117, 123, 128);">[Optional]</span> <b>Modify the environment</b> of the MPI processes before launching, such as prepending to LD_PRELOAD or LD_LIBRARY_PATH to inject preloads into the MPI processes (but not other processes, such
as MPI daemons or tool daemons).<br>
Q: When? It would be great if modifying the environment could be done just before the MPI processes are forked/exec’ed. Why? For our profiler MAP we need to preload an MPI specific PMPI library, but for some “MPI” starter processes such as srun we don’t know
the real MPI implementation that is used. Acquiring a partial process table (no PIDs) and inspecting the binaries pointed to by the executable paths, could allow us to detect the MPI implementation.<br>
</li><li><span style="color: rgb(117, 123, 128);">[Optional]</span> <b>Raise queue allocation request and granted events</b> to allow the tool to disable startup timeouts temporarily while a queue allocation is in progress, as this can take a long time (applicable
to srun). </li><li><span style="color: rgb(117, 123, 128);">[Optional]</span> <b>Scalable startup of tool daemons and shipping of preloads</b> without touching the parallel file system on compute nodes. For example, pushing of daemon executable file (and dependencies plus
preloads) to ram disk on compute nodes or Spindle?<br>
Q: Where files are pushed? A job specific temporary directory such as /tmp/mpi.job.1234/... or configurable?<br>
Q: How can the tool query the location of pushed files to reference them?<br>
Q: Who cleans up the files afterwards? The MPI job/starter itself or the tool?<br>
N: Files should be pushed with original file’s permissions. </li></ul>
<br>
NB: Optional requirement = A tool could ultimately work without it, but I very much would like to see it supported.
<br>
<br>
In PMIx this is the “indirect” use case and Ralph Castain and I have been working together since a while to make sure this use case will be supported by PMIx (excluding some optional requirements for now).<br>
</div>
<div><br>
<i>Additional questions/notes:</i> <br>
<br>
<ul style="margin-bottom: 0px; margin-top: 0px;">
<li>How will the handshake work, such that the MPI starter knows it's running under a tool and when it's possible for a tool to interact with the MPI starter process?
</li><li>The handshake must support cases where the MPI starter filename that is provided by the user is a wrapper around the real MPI starter executable, for example XALT. </li><li>In PMIx both problems are solved with PMIX_LAUNCHER_PAUSE_FOR_TOOL=1 and PMIX_LAUNCHER_RENDEZVOUS_FILE=<filename>.</li><li>Will only the tool or also the tool daemons interact with the MPI job?</li><li>Security, for example USER B shall not be able to attach to MPI job of USER A.</li><li>Anything to consider for MPMD or heterogeneous systems?</li><li>Must be scalable.</li><li>Must not require a debugger. <br>
</li></ul>
<br>
<b>Use case 2: Attach to a running MPI job with a tool. </b><br>
<br>
This use case is a subset of use case 1 with the following requirements: <br>
<br>
<ul style="margin-bottom: 0px; margin-top: 0px;">
<li>Acquire the process table of spawned MPI processes of MPI job. </li><li>Spawn tool daemon on nodes where MPI job is running. </li><li><span style="color: rgb(117, 123, 128);">[Optional]</span> Scalable startup of tool daemons without touching the file system.
</li></ul>
</div>
<br>
<p></p>
<p style="margin-top:0;margin-bottom:0">Thanks,</p>
<p style="margin-top:0;margin-bottom:0">Dirk<br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div><span style="font-size:10pt"><b>Dirk Schubert | Arm |</b> Staff Software Engineer </span></div>
<div><span style="font-size:10pt">dirk.schubert@arm.com</span></div>
<div><span style="font-size:10pt">
<p style="margin-bottom:0cm; line-height:100%"><b><font color="#3a5979">allinea</font>
</b><font color="#808080">is now part of</font><b><font color="#808080"> </font><font color="#138dac">Arm</font></b></p>
</span></div>
</div>
</div>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> mpiwg-tools <mpiwg-tools-bounces@lists.mpi-forum.org> on behalf of Mohror, Kathryn via mpiwg-tools <mpiwg-tools@lists.mpi-forum.org><br>
<b>Sent:</b> 17 March 2019 14:46<br>
<b>To:</b> mpiwg-tools@lists.mpi-forum.org<br>
<b>Cc:</b> Mohror, Kathryn<br>
<b>Subject:</b> [mpiwg-tools] Meeting March 21, 2019</font>
<div> </div>
</div>
<div link="#0563C1" vlink="#954F72" lang="EN-US">
<div class="x_WordSection1">
<p class="x_MsoNormal">Hey everybody,</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">For our call this Thursday (3/21) we’ll come back to debugger topics again. The call is at the usual time in the US (8 am Pacific/ 11 am Eastern / 4 pm CET) but note that the US has moved to DST but I don’t think the EU has done so yet.
</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Webex Info:</p>
<p class="x_MsoNormal"> https://apps.fz-juelich.de/scalasca/mpiwg-tools-webex/index.php</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Agenda:</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">In the last debugger call, I said that the plan for this meeting would be to get an overview of OMPD and start talking about analogous interfaces for MPI (e.g. revamp MQD). However, in the meantime, I was convinced that we should focus
on process acquisition first. So, the plan will be to talk about process acquisition in this meeting. I’ll go through our notes from 2017(ish) to hopefully find the straw men we drafted back then. Please bring your use cases and any ideas you have on this
front.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">_________________________________________________________________</p>
<p class="x_MsoNormal">Kathryn Mohror, <a href="mailto:kathryn@llnl.gov"><span style="color:blue">kathryn@llnl.gov</span></a>,
<a href="https://people.llnl.gov/kathryn"><span style="color:blue">https://people.llnl.gov/kathryn</span></a></p>
<p class="x_MsoNormal"><a href="https://computation.llnl.gov/casc/data-analysis-group"><span style="color:blue">Data Analysis Group</span></a> @ Lawrence Livermore National Laboratory, Livermore, CA, USA</p>
<p class="x_MsoNormal"> </p>
</div>
</div>
</div>
</div>
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose,
or store or copy the information in any medium. Thank you.
</body>
</html>