<div dir="ltr">MPI has not been a message passing library since MPI-1. It's a runtime system for HPC that provides interprocess communication of many kinds (not just message passing - see RMA) as well as parallel file I/O.<div><br></div><div>Jeff</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 20, 2016 at 3:06 PM, Teranishi, Keita <span dir="ltr"><<a href="mailto:knteran@sandia.gov" target="_blank">knteran@sandia.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="m_-7925476064558468283WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">All,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Throughout the discussion, I am a bit worried about making MPI bigger than message passing interface because I wish MPI to serve a good abstraction of user-friendly transport layer. Fenix
is intended to leverage the minimalist approach of MPI-FT (ULFM today) to cover most of online recovery models for parallel programs using MPI. The current version is designed to support SPMD (Communicating Sequential Process) model, but we wish to support
other models including Master-Worker, Distributed Asynchronous Many Task (AMT) and Message-Logging.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="m_-7925476064558468283MsoListParagraph"><u></u><span style="font-size:11.0pt;font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:Calibri">ULFM: We have requested non-blocking communicator recovery as well as non-blocking comm_dup and comm_split, etc. ULFM already provides good mechanism to serve master-worker
type recovery like UQ, model reduction and a certain family of eigenvalue solvers. I wish to have more fine control for revocation because it is possible to keep the certain connection of survived process (for master-worker or task-parallel computing), but
it might be too difficult.<u></u><u></u></span></p>
<p class="m_-7925476064558468283MsoListParagraph"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="m_-7925476064558468283MsoListParagraph"><u></u><span style="font-size:11.0pt;font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:Calibri">ULFM + Auto recovery: I need clarification from Wesly (as my knowledge is wrong most likely… but let me continue based on my assumption). Fenix assumes that failure happens at
a single or a small number of processes. In this model, auto-recovery could serve as un-coordinated recovery because no comm_shrink call is used to fix the communicator. This could help message reply of uncoordinated recovery model. For example, recovery
is never manifested as “Failure” to the survived ranks, making particular message passing calls very slow. For SPMD model, adaptation is so challenging as the user needs to write how to recover the lost state of failed processes. However, I can see a great
benefit for implementing resilient task parallel programming model. <u></u><u></u></span></p>
<p class="m_-7925476064558468283MsoListParagraph"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="m_-7925476064558468283MsoListParagraph"><u></u><span style="font-size:11.0pt;font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:Calibri">Communicator with hole: Master-Worker type applications will benefit from this when making collectives to gather the data available.<u></u><u></u></span></p>
<p class="m_-7925476064558468283MsoListParagraph"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="m_-7925476064558468283MsoListParagraph"><u></u><span style="font-size:11.0pt;font-family:Symbol"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:Calibri">MPI_ReInit: MPI_ReInit is very close to the current Fenix model. We have written the API specification (see attached) to support the same type of online recovery (global rollback
upon process failure). The code is implemented using MPI-ULFM, and we have seen some issues with MPI-ULFM that makes multiple communicator recovery convoluted. We used PMPI to hide all the details of error handling, garbage collection and communicator recovery.
The rollback (to Fenix_Init) is performed through longjmp. Nice features of Fenix are (1) an idea of
<b>resilient communicator</b> that allows the users to specify which communicator needs to be automatically fixed and (2)
<b>callback functions</b> to assist application-specific recovery followed by communicator recovery. We originally do not intend Fenix to be part of the MPI standard because we want the role of MPI confined within “Message Passing” and do not want delay the
MPI standardization discussions. My understanding with MPI_ReInit is standardizing online-rollback recovery and keeping PMPI/QMPI layer clean through a tight binding with the layers invisible to typical MPI users (or tool developers) --- Ignacio, please
correct me if I am wrong. My biggest concern of MPI_ReInit is that defining rollback model by Message Passing Library may violate the original design philosophy of MPI (again this is the reason why we did not propose Fenix as MPI standard). Another concern
is that it might be difficult to keep other recovery options open, but it gets much more flexible with a few knobs in the APIs. I think the latter is easy to fix with some switches in APIs. I think we can figure out the options as we discuss further.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Keita<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">"Bland, Wesley" <<a href="mailto:wesley.bland@intel.com" target="_blank">wesley.bland@intel.com</a>><br>
<b>Date: </b>Tuesday, December 20, 2016 at 1:48 PM</span></p><div><div class="h5"><br>
<b>To: </b>MPI WG Fault Tolerance and Dynamic Process Control working Group <<a href="mailto:mpiwg-ft@lists.mpi-forum.org" target="_blank">mpiwg-ft@lists.mpi-forum.org</a>><wbr>, "Teranishi, Keita" <<a href="mailto:knteran@sandia.gov" target="_blank">knteran@sandia.gov</a>><br>
<b>Subject: </b>Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today<u></u><u></u></div></div><p></p>
</div><div><div class="h5">
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:Helvetica">Probably here since we don't have an issue for this discussion. If you want to open issues in our working group's repository (<a href="http://github.com/mpiwg-ft/ft-issues" target="_blank">github.com/mpiwg-ft/ft-issues</a><wbr>), that's
probably fine.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<p class="m_-7925476064558468283airmailon">On December 20, 2016 at 3:47:25 PM, Teranishi, Keita (<a href="mailto:knteran@sandia.gov" target="_blank">knteran@sandia.gov</a>) wrote:<u></u><u></u></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Wesley,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Should I do here or github issues?
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Thanks,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Keita</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From:
</span></b><span style="font-family:Calibri;color:black">"Bland, Wesley" <<a href="mailto:wesley.bland@intel.com" target="_blank">wesley.bland@intel.com</a>><br>
<b>Date: </b>Tuesday, December 20, 2016 at 1:43 PM<br>
<b>To: </b>MPI WG Fault Tolerance and Dynamic Process Control working Group <<a href="mailto:mpiwg-ft@lists.mpi-forum.org" target="_blank">mpiwg-ft@lists.mpi-forum.org</a>><wbr>, "Teranishi, Keita" <<a href="mailto:knteran@sandia.gov" target="_blank">knteran@sandia.gov</a>><br>
<b>Subject: </b>Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:Helvetica">You don't have to wait. :) If you have comments/concerns, you can raise them here too.</span><u></u><u></u></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"> <u></u><u></u></p>
<p class="m_-7925476064558468283airmailon0">On December 20, 2016 at 3:38:47 PM, Teranishi, Keita (<a href="mailto:knteran@sandia.gov" target="_blank">knteran@sandia.gov</a>) wrote:<u></u><u></u></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">All,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Sorry, I could not make it today. I will definitely join the meeting next time to make comments/suggestions on the three items
(ULFM, ULFM+Auto, and ReInit) from Fenix perspective.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Thanks,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Keita</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From:
</span></b><span style="font-family:Calibri;color:black"><<a href="mailto:mpiwg-ft-bounces@lists.mpi-forum.org" target="_blank">mpiwg-ft-bounces@lists.mpi-<wbr>forum.org</a>> on behalf of "Bland, Wesley" <<a href="mailto:wesley.bland@intel.com" target="_blank">wesley.bland@intel.com</a>><br>
<b>Reply-To: </b>MPI WG Fault Tolerance and Dynamic Process Control working Group <<a href="mailto:mpiwg-ft@lists.mpi-forum.org" target="_blank">mpiwg-ft@lists.mpi-forum.org</a>><br>
<b>Date: </b>Tuesday, December 20, 2016 at 1:29 PM<br>
<b>To: </b>FTWG <<a href="mailto:mpiwg-ft@lists.mpi-forum.org" target="_blank">mpiwg-ft@lists.mpi-forum.org</a>><br>
<b>Subject: </b>[EXTERNAL] Re: [mpiwg-ft] FTWG Con Call Today</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">The notes from today's call are posted on the wiki:</span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica"> </span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><a href="https://github.com/mpiwg-ft/ft-issues/wiki/2016-12-20" target="_blank">https://github.com/mpiwg-ft/<wbr>ft-issues/wiki/2016-12-20</a><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal">Those who have specific items, please make progress on those between now and our next meeting. We will be cancelling the Jan 3 call due to the holiday. The next call will be on
Jan 17.<u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal">Wesley<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="m_-7925476064558468283airmailon00">On December 20, 2016 at 8:15:06 AM, Bland, Wesley (<a href="mailto:wesley.bland@intel.com" target="_blank">wesley.bland@intel.com</a>) wrote:<u></u><u></u></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica"> </span><u></u><u></u></p>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM Eastern. Today's agenda:</span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica"> </span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">* Recap of face to face meeting</span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">* Go over existing tickets</span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">* Discuss concerns with ULFM and path forward</span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica"> </span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">Thanks, </span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">Wesley </span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica"> </span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">..............................<wbr>..............................<wbr>..............................<wbr>..............................<wbr>.................</span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">Join online meeting <<a href="https://meet.intel.com/wesley.bland/GHHKQ79Y" target="_blank">https://meet.intel.com/<wbr>wesley.bland/GHHKQ79Y</a>></span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica"><a href="https://meet.intel.com/wesley.bland/GHHKQ79Y" target="_blank">https://meet.intel.com/wesley.<wbr>bland/GHHKQ79Y</a></span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica"> </span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">Join by Phone</span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">+1(916)356-2663 (or your local bridge access #) Choose bridge 5.</span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">Find a local number <<a href="https://dial.intel.com" target="_blank">https://dial.intel.com</a>></span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica"> </span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">Conference ID: 757343533</span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica"> </span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">Forgot your dial-in PIN? <<a href="https://dial.intel.com" target="_blank">https://dial.intel.com</a>> | First online meeting? <<a href="http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4&p2=1041&pc=oc&ver=4&subver=0&bld=7185&bldver=0" target="_blank">http://r.office.microsoft.<wbr>com/r/rlidOC10?clid=1033&p1=4&<wbr>p2=1041&pc=oc&ver=4&subver=0&<wbr>bld=7185&bldver=0</a>></span><u></u><u></u></p>
</div>
<div id="m_-7925476064558468283bloop_customfont">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:Helvetica">..............................<wbr>..............................<wbr>..............................<wbr>..............................<wbr>.................</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal">______________________________<wbr>_________________
<br>
mpiwg-ft mailing list <br>
<a href="mailto:mpiwg-ft@lists.mpi-forum.org" target="_blank">mpiwg-ft@lists.mpi-forum.org</a> <br>
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft" target="_blank">https://lists.mpi-forum.org/<wbr>mailman/listinfo/mpiwg-ft</a><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div></div></div>
</div>
<br>______________________________<wbr>_________________<br>
mpiwg-ft mailing list<br>
<a href="mailto:mpiwg-ft@lists.mpi-forum.org">mpiwg-ft@lists.mpi-forum.org</a><br>
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft" rel="noreferrer" target="_blank">https://lists.mpi-forum.org/<wbr>mailman/listinfo/mpiwg-ft</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div>