<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">IIRC, there may be a default range setting that blocks the lookup - I’ll try to take a look later. Please understand that this isn’t an operation we spend a lot of effort supporting as virtually nobody has used it.<div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 20, 2018, at 4:26 PM, HOLMES Daniel <<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Ralph,
<div class=""><br class="">
</div>
<div class="">Unfortunately, the fact that “test3” is passing the “portname" on the command-line is a red-herring - connect/accept also fails when the “portname” is advertised by the job that opens the port using MPI_Publish_name and discovered by the other
job using MPI_Lookup_name. This is the test case that the sandbox code relies on. I also modified the “test3” example to use Publish/Lookup to figure out if there was any difference in the internal handling (inside Open MPI and/or inside PMIx) between these
situations. There is no difference in the final outcome or in the code path from dpm_connect_accept to the deadlock.<br class="">
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="">
</div>
<div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">In
all cases, both processes get as far as ompi/dpm/dpm.c:398 (using git commit 5768336) and calls into PMIx_Connect. They both then get as far as opal/mca/pmix/pmix3x/pmix/src/client/pmix_client_connect.c:102 (same git hash), i.e.
</span>PMIX_WAIT_THREAD(&cb->lock);</div>
<div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
</div>
<div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
What happens next is an infinite loop that (at least) includes a whole bunch of calls to "HASH:STORE rank -2 key pmix.<various>” output messages that burns a couple of CPUs at 100% and shifts data on the local network at max-bandwidth.</div>
<div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
</div>
<div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
The MacOS Activity Monitor shows two orterun processes at 50% CPU each and one orte-server process at 100%. Network usage statistics are roughly in the same proportion.</div>
<div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
</div>
<div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
It’s going to be hard to progress any further without knowing a starting point for the code doing the HASH:STORE operations. I guess it’s the PMIx progress threads trying to complete the PMIX_PTL_SEND_RECV operations pushed onto the event queues in PMIx_Connect_nb
but that isn’t helping me all that much.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="Apple-interchange-newline">
Cheers,</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Dan.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Applications Consultant in HPC Research<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—</div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On 20 Aug 2018, at 16:24, Ralph H Castain <<a href="mailto:rhc@open-mpi.org" class="">rhc@open-mpi.org</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Passing a port on the cmd line for accept/connect was never implemented as I don’t think anyone really cared. Given how OMPI uses PMIx for that operation, it shouldn’t be all that difficult to do.
<div class=""><br class="">
</div>
<div class="">As noted in the referenced issue, there was a problem last year with cross-mpirun connections. Not sure when I’ll have time to look at it.</div>
<div class=""><br class="">
</div>
<div class="">Canceling the meeting today is fine with me - I got pulled away and didn’t get the PMIx Groups implementation done (sigh).</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 20, 2018, at 8:10 AM, HOLMES Daniel <<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Hi Howard,
<div class=""><br class="">
</div>
<div class="">Thanks for the update. Sounds promising.</div>
<div class=""><br class="">
</div>
<div class="">I'm trying to fix the test3.zip example from:</div>
<div class=""><a href="https://github.com/open-mpi/ompi/issues/3458#issuecomment-322951227" class="">https://github.com/open-mpi/ompi/issues/3458#issuecomment-322951227</a><br class="">
<div class=""><br class="webkit-block-placeholder">
</div>
<div class="">If successful, this would extend the testing opportunities for the sandbox code to situations that involve more than one mpirun. The issue is definitely some sort of deadlock in PMIx but I’ve not figured it out completely yet.</div>
<div class=""><br class="webkit-block-placeholder">
</div>
<div class="">
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
I’m cancelling the meeting today, unless anyone objects in the next 50 minutes.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="Apple-interchange-newline">
Cheers,</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Dan.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Applications Consultant in HPC Research<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—</div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On 20 Aug 2018, at 15:55, Pritchard Jr., Howard <<a href="mailto:howardp@lanl.gov" class="">howardp@lanl.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Calibri, sans-serif;" class="">
<div class="">HI Folks,</div>
<div class=""><br class="">
</div>
<div class="">I’ll be on a plane at 11 AM MDT today so will not be able to call in.</div>
<div class=""><br class="">
</div>
<div class="">I tried running the tests Dan had added/modified and observed</div>
<div class="">what he did, that one can’t allow more than one outstanding</div>
<div class="">accept/connect going on at a time or Open MPI’s ORTE gets confused.</div>
<div class="">I reduced this down to a simpler test which hangs with only 3 ranks</div>
<div class="">and am narrowing down what the issue is.</div>
<div class=""><br class="">
</div>
<div class="">I’ll be opening a PR with changes to chapter 8 of the standard and</div>
<div class="">replacement for MPI_Get_Set_Names later this week.</div>
<div class=""><br class="">
</div>
<div class="">Howard</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div class="">-- </div>
<div class="">Howard Pritchard</div>
</div>
<div class="">B Schedule</div>
<div class="">HPC-ENV</div>
<div class="">
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt;" class="">Office 9, 2<sup class="">nd</sup> floor Research Park<o:p class=""></o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 11pt;" class="">TA-03, Building 4200, Room 203</div>
</div>
<div class="">Los Alamos National Laboratory</div>
<div class=""><br class="">
</div>
</div>
</div>
_______________________________________________<br class="">
mpiwg-sessions mailing list<br class="">
<a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
The University of Edinburgh is a charitable body, registered in<br class="">
Scotland, with registration number SC005336.<br class="">
_______________________________________________<br class="">
mpiwg-sessions mailing list<br class="">
<a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
mpiwg-sessions mailing list<br class="">
<a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a><br class="">
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions" class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
The University of Edinburgh is a charitable body, registered in<br class="">Scotland, with registration number SC005336.<br class="">_______________________________________________<br class="">mpiwg-sessions mailing list<br class=""><a href="mailto:mpiwg-sessions@lists.mpi-forum.org" class="">mpiwg-sessions@lists.mpi-forum.org</a><br class="">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions<br class=""></div></blockquote></div><br class=""></div></body></html>