<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="orphans: auto; text-align: start; text-indent: 0px; widows: auto; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; orphans: 2; text-align: -webkit-auto; text-indent: 0px; widows: 2; border-spacing: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; orphans: 2; text-align: -webkit-auto; text-indent: 0px; widows: 2; border-spacing: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; orphans: 2; text-align: -webkit-auto; text-indent: 0px; widows: 2; border-spacing: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; orphans: 2; text-indent: 0px; widows: 2; border-spacing: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; orphans: 2; text-indent: 0px; widows: 2; border-spacing: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; orphans: 2; text-indent: 0px; widows: 2; border-spacing: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><font face="Lucida Grande" class="">--</font></div><div style="color: rgb(0, 0, 0); font-family: 'Lucida Grande'; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;" class=""><span style="text-align: -webkit-auto;" class="">Aurélien Bouteiller, Ph.D. ~~ </span><span style="text-align: -webkit-auto;" class=""><a href="https://icl.cs.utk.edu/~bouteill/" class="">https://icl.cs.utk.edu/~bouteill/</a></span></div></div></span></div></span></div></span></div></span></div></span></div></span></div></div></div></div></div></div></div></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">Le 14 déc. 2015 à 09:19, Jeff Hammond <<a href="mailto:jeff.science@gmail.com" class="">jeff.science@gmail.com</a>> a écrit :</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><br class="Apple-interchange-newline"><br class=""><div class="gmail_quote">On Fri, Dec 11, 2015 at 2:09 PM, Bland, Wesley<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:wesley.bland@intel.com" target="_blank" class="">wesley.bland@intel.com</a>></span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">Hi WG,<br class=""><br class="">I’ve put together some notes on the goings on around our working group at the forum. You can find them all on the wiki page:<br class=""><br class=""><a href="https://github.com/mpiwg-ft/ft-issues/wiki/2015-12-07" rel="noreferrer" target="_blank" class="">https://github.com/mpiwg-ft/ft-issues/wiki/2015-12-07</a><br class=""><br class="">Since I know that the click-through is not always practical, I’ll copy them below.<br class=""><br class="">Thanks,<br class="">Wesley<br class=""><br class="">====<br class=""><br class="">WG Meeting<br class=""><br class=""> <span class="Apple-converted-space"> </span>* Went over reading for plenary time<br class=""> <span class="Apple-converted-space"> </span>* Aurelien and Keita presented some of the results of the ULFM BoF at SC<br class=""> * Attendance was great<br class=""> * There were a few questions and suggestions to improve the proposal.<br class=""> <span class="Apple-converted-space"> </span>* Aurelien is creating issues for suggests that we will act on.<br class=""> <span class="Apple-converted-space"> </span>* We discussed an overall view of what fault tolerance and error handling means in the context of MPI and how we cover each area as a standard<br class=""> * We divided applications into a few buckets:<br class=""> <span class="Apple-converted-space"> </span>* Current applications - This describes the vast majority of applications which require that all process remain alive and recovery tends to be global.<br class=""> * These apps tend to use/require recovery very similar to checkpoint/restart.<br class=""> * They probably don't derive a lot of benefit from ULFM-like recovery models, but could potentially benefit from improved error handlers.<br class=""></blockquote><div class=""><br class=""></div><div class="">Just remember that there can be ULFM + { hot spares -or- MPI_Comm_spawn(_multiple) } + checkpoint-restart, which preserves the size of the job.</div><div class=""><br class=""></div></div></div></div></div></blockquote><div>Totally agree with you. Some point was made that in some of these I/O based checkpointing apps, existing MPI standard, or only slightly modified MPI standard could be enough, which is also true (as long as you remain on the I/O based checkpointing). </div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class="">Lots of those fault-intolerant apps can do something like this with minimal changes. The apps that cannot use ULFM are the ones that require expensive message logging to be able to roll-back to a coherent state.</div></div></div></div></div></blockquote><div><br class=""></div><div>There has been some work to provided automatic C/R with message logging with ULFM, so that’s also possible (the paper claims to beat our own implementation of ML in Open MPI, when it compares to O2P, and that’s actually pretty good, if experiments are conducted correctly, because O2P is fast).</div><div><a href="http://link.springer.com/chapter/10.1007%2F978-3-319-03859-9_27" class="">http://link.springer.com/chapter/10.1007%2F978-3-319-03859-9_27</a></div><div><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"> <span class="Apple-converted-space"> </span>* In-memory Checkpoint/Restart - These apps can use in-memory checkpointing to improve both checkpoint and recovery times. They usually need to replace failed processes, but don't require that all remain alive.<br class=""> * ULFM is a possibility here, but can result in bad locality without a library which will automatically move processes around after a failure.<br class=""></blockquote><div class=""><br class=""></div><div class="">Maybe. I'm not sure good locality exists on fat-tree and dragonfly topologies. And users can always renumber their ranks and move data around if they know topological placement matters.</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"> * Reinit / multi-init/finalize with improved PMPI would also work. There are some proposals going on or that have gone on which could also provide the needed functionality. In these proposals, most of the locality problems would probably be pushed into the MPI library when initialized again.<br class=""> <span class="Apple-converted-space"> </span>* New applications - These apps tend to be able to run with fewer processes. They cover apps like tasking models, master/worker apps, and traditionally non-MPI apps that might be interested in the future (Hadoop, etc.).<br class=""></blockquote><div class=""><br class=""></div><div class="">Lots of current applications are master-worker...</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"> * ULFM generally would apply well to these applications as locality is less important if processes are not being replaced.<br class=""> * There are also errors that do not include process failures:<br class=""> <span class="Apple-converted-space"> </span>* Memory errors<br class=""></blockquote><div class=""><br class=""></div><div class="">It is useful to distinguish causes and effects here. Memory errors are a cause. Process failure is one effect. Another effect is non-fatal data corruption, which may or may not be silent. Currently, we see that memory errors that are detected manifest as process failures, but hopefully someday the OS/RT people will figure out better things to do than just call abort(). Ok, to be fair, it's probably the firmware/driver people throwing the self-destruct lever...</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"> * These could be detected by anything, but ULFM revoke could help with notification.<br class=""> * Lots of SDC research is out there that sits on top of MPI.<br class=""> <span class="Apple-converted-space"> </span>* Network errors<br class=""></blockquote><div class=""><br class=""></div><div class="">I am curious how often this actually happens anymore. Aren't all modern networks capable of routing around permanently failed links? A switch failure should be fatal but that happens how often?</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"> * These tend to be masked by the implementation or promoted to process failures<br class=""> <span class="Apple-converted-space"> </span>* Resource exhaustion<br class=""> * These sorts of errors cover out of memory, out of context IDs, etc.<br class=""> * They can be improved with better error handlers/codes/classes<br class=""></blockquote><div class=""><br class=""></div><div class="">Yes, and this will be wildly useful. Lots of users lose jobs because they do dumb stuff with communicators that could be mitigated with slower fallback implementations over p2p (please ignore the apparent false-dichotomy here).</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"> <span class="Apple-converted-space"> </span>* Discussed some new topics related to error handling and error codes/classes<br class=""> * Pavan expressed interest in error codes saying whether they were catastrophic or not.<br class=""> <span class="Apple-converted-space"> </span>* This resulted mpi-forum/mpi-issues#28<<a href="https://github.com/mpi-forum/mpi-issues/issues/28" rel="noreferrer" target="_blank" class="">https://github.com/mpi-forum/mpi-issues/issues/28</a>> where we add a new call MPI_ERROR_IS_CATASTROPHIC.<br class=""><br class=""><<a href="https://github.com/mpiwg-ft/ft-issues/wiki/2015-12-07#plenary-time" rel="noreferrer" target="_blank" class="">https://github.com/mpiwg-ft/ft-issues/wiki/2015-12-07#plenary-time</a>>Plenary Time<br class=""><br class=""> <span class="Apple-converted-space"> </span>* Read the error handler cleanup tickets mpi-forum/mpi-issues#1<<a href="https://github.com/mpi-forum/mpi-issues/issues/1" rel="noreferrer" target="_blank" class="">https://github.com/mpi-forum/mpi-issues/issues/1</a>> and mpi-forum/mpi-issues#3<<a href="https://github.com/mpi-forum/mpi-issues/issues/3" rel="noreferrer" target="_blank" class="">https://github.com/mpi-forum/mpi-issues/issues/3</a>>.<br class=""> * The forum didn't like where we removed all of the text about general errors. They considered some of it to still be valuable and should be updated. In particular, the example about MPI_RSEND could still be applicable if the implementation decides that it wants to return an error to the user because the MPI_RECV was not posted.<br class=""> * We need to add text for MPI_INTERCOMM_CREATE.<br class=""> * A few other minor things were added directly to the pull request.<br class=""> <span class="Apple-converted-space"> </span>* Read the MPI_COMM_FREE advice ticket.<br class=""> * No concerns, will vote at next meeting.<br class=""> <span class="Apple-converted-space"> </span>* Presented the plenary about catastrophic errors.<br class=""> * Few concerns were raised during the plenary. The main one was from Bill who says we should look at how other standards describe non-fatal errors when writing the text here.<br class=""></blockquote><div class=""><br class=""></div><div class="">I'm skeptical that this is going to help us, but here are some references:</div><div class="">- Fortran 2008 section 14.6 Halting (not going to be useful to us, although its utility for users is demonstrated in NOTE 14.16); section 2.3.5 describes how any error propagates to all images (in a coarray program). There is no notion of fault-tolerance here.</div><div class="">- UPC and OpenSHMEM say nothing about fault-tolerance. I suspect that UPC never will and OpenSHMEM will try to learn from the MPI Forum.</div><div class="">- C++14 draft (N3936) chapter 19 is all about exceptions and errors. 30.2.2 talks about thread failure.</div><div class="">- IB 1.0 7.12.2 and 7.12.3, among other places.</div><div class=""><br class=""></div><div class="">Most specifications that I've read don't have all of the baggage we do, but only because most of their objects are not so stateful or long-lived. Now, if MPI was based upon connections rather than communicators…</div></div></div></div></div></blockquote><div><br class=""></div><div>Thanks for the links, very useful! </div><div><br class=""></div><div>Aurélien</div><div><br class=""></div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"> * Ryan asked about the general usefulness of this proposal in terms of how an application would be able to respond to information about whether an error is fatal or not.<br class=""> <span class="Apple-converted-space"> </span>* He asserts that error classes should generally be descriptive enough without it and if they aren't, the error class itself should be improved.<br class=""></blockquote><div class=""><br class=""></div><div class="">Best,</div><div class=""><br class=""></div><div class="">Jeff</div><div class=""> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">_______________________________________________<br class="">mpiwg-ft mailing list<br class=""><a href="mailto:mpiwg-ft@lists.mpi-forum.org" class="">mpiwg-ft@lists.mpi-forum.org</a><br class=""><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft" rel="noreferrer" target="_blank" class="">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft</a></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>--<span class="Apple-converted-space"> </span><br class=""><div class="gmail_signature">Jeff Hammond<br class=""><a href="mailto:jeff.science@gmail.com" target="_blank" class="">jeff.science@gmail.com</a><br class=""><a href="http://jeffhammond.github.io/" target="_blank" class="">http://jeffhammond.github.io/</a></div></div></div><span style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">mpiwg-ft mailing list</span><br style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:mpiwg-ft@lists.mpi-forum.org" style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">mpiwg-ft@lists.mpi-forum.org</a><br style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft" style="font-family: Menlo-Regular; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft</a></div></blockquote></div><br class=""></body></html>