[Mpi3-ft] Ticket 323 - status?

Thu May 31 07:32:34 CDT 2012

Ralph,

You site a published study. Can you provide a link to the resource?

-- Josh

On Wed, May 30, 2012 at 10:18 PM, Ralph Castain <rhc at open-mpi.org> wrote:
> Obviously, I can't speak for the folks who attended and voted "no", either
> directly or by abstaining. However, I have talked to at least a few people,
> and can offer a point or two about the concerns.
>
> First, the last study I saw published on the subject of FT for MPI showed a
> very low level of interest in FT within the MPI community. It based this on
> a usage analysis that showed something over 90% of clusters being too small
> to see large failure rates. On the clusters that were large enough
> (primarily at the national labs, who pretty clearly voted no), over 80% of
> the MPI jobs lasted less than 1 hour.
>
> So the size of the community that potentially benefits from FT is very
> small. In contrast, despite assurances it would be turned off unless
> specifically requested, it was clear from the proposals that FT would impact
> a significant fraction of the code, thus raising the potential for a
> substantial round of debugging and instability.
>
> For that majority who would see little-to-no benefit, this isn't an
> attractive trade-off.
>
> Second, those who possibly could benefit tend to take a more holistic view
> of FT. If you step back and look at the cluster as a system, then there are
> multiple ways of addressing the problems of failure during long runs. Yes,
> one way is to harden MPI to such events, but that is probably the hardest
> solution.
>
> One easier way, and the one being largely touted at the moment, is to make
> checkpointing of an application be a relatively low-cost event so that it
> can be frequently done. This is being commercialized as we speak by the
> addition of SSDs to the usual parallel file system, thus making a checkpoint
> run at very fast speeds. In fact, "burst" buffers are allowing the
> checkpoint to dump very quickly, and then slowly drain to disk, rendering
> the checkpoint operation very low cost. Given that the commercial interests
> coincide with the HPC interests, this solution is likely to be available
> from cluster suppliers very soon at an attractive price.
>
> Combined with measures to make restart very fast as well, this looks like an
> alternative that has no performance impact on the application at the MPI
> level, doesn't potentially destabilize the software, and may meet the
> majority of needs.
>
> I'm not touting this approach over any other, mind you - just trying to
> point out that the research interests of the FT/MPI group needs to be
> considered in a separate light from the production interests of the
> community. What you may be experiencing (from my limited survey) is a
> reflection of that divergence.
>
> Ralph
>
>
>
> On Wed, May 30, 2012 at 6:55 PM, George Bosilca <bosilca at eecs.utk.edu>
> wrote:
>>
>> On May 31, 2012, at 08:44 , Martin Schulz wrote:
>>
>> Several people who abstained had very similar concerns, but chose the
>> abstain vote since they know it meant no,
>>
>>
>> Your interpretation is barely making a "simple majority" in the forum, as
>> highlighted by parallel discussions in the other email threads. But let's
>> leave this discussion in its own thread.
>>
>> But you're right, both "no" and "abstain" votes should be addressed. Bill
>> made his point very clear, and to be honest he was the only one that raised
>> a __valid__ point about the FT proposal. Personally, I am looking forward to
>> fruitful discussions during our weekly phone-calls where the complaints
>> raised during the voting will be brought forward in the way that the working
>> group will have a real opportunity to address them as they deserve. In other
>> terms we are all counting on you guys to enlighten us on the major issues
>> with this proposal and the potential solutions you envision or promote.
>>
>>   george.
>>
>> On May 31, 2012, at 08:44 , Martin Schulz wrote:
>>
>> Hi George,
>>
>> One other no was Intel as far as I remember, but I don't remember the 5th.
>> However, I would suggest not to focus on the no votes alone. Several people
>> who abstained had very similar concerns, but chose the abstain vote since
>> they know it meant no, but they agreed with the general necessity of FT for
>> MPI. I remember, for example, Bill saying that for him abstain meant no, but
>> that changes later on could change his mind. Based on this interpretation,
>> the ticket definitely had more than 5 no votes.
>>
>> Martin
>>
>>
>> On May 31, 2012, at 8:34 AM, Darius Buntinas wrote:
>>
>>
>> Argonne was not convinced that we (FTWG) had the right solution, and the
>> large changes in the text mentioned previously did not instill confidence.
>>  So it was decided that Argonne would vote against the ticket.
>>
>> -d
>>
>> On May 30, 2012, at 6:24 PM, George Bosilca wrote:
>>
>> In total there were 5 no votes. I wonder who were the other two, they
>> might be willing to enlighten us on their reasons to vote against.
>>
>>
>> george.
>>
>>
>> On May 31, 2012, at 05:48 , Anthony Skjellum wrote:
>>
>>
>> Three no votes were LLNL, Argonne, and Sandia.  Since MPI is heavily
>> driven by DOE, convincing these folks would be important.
>>
>>
>> Tony Skjellum, tonyskj at yahoo.com or skjellum at gmail.com
>>
>> Cell 205-807-4968
>>
>>
>> On May 31, 2012, at 5:10 AM, Richard Graham <richardg at mellanox.com> wrote:
>>
>>
>> The main objection raised is that the text has still been having large
>> changes, and if not for the pressure of the 3.0 deadline, this would not
>> have come up for a vote.  I talked one-on-one with many that either voted
>> against or abstained, and this was the major (not only) point raised.
>>
>>
>> Rich
>>
>>
>> -----Original Message-----
>>
>> From: mpi3-ft-bounces at lists.mpi-forum.org
>> [mailto:mpi3-ft-bounces at lists.mpi-forum.org] On Behalf Of Aurélien
>> Bouteiller
>>
>> Sent: Wednesday, May 30, 2012 10:05 PM
>>
>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group
>>
>> Subject: Re: [Mpi3-ft] Ticket 323 - status?
>>
>>
>> It seems we had very little, if any, technical opposition on the content
>> of the proposal itself, but mostly comments on the process. I think we need
>> to understand more what are the oppositions. Do we have a list of who voted
>> for and against and their rationale?
>>
>>
>> Aurelien
>>
>>
>>
>> Le 30 mai 2012 à 08:52, Josh Hursey a écrit :
>>
>>
>> That is unfortunate. A close vote (7 yes to 9 no/abstain). :/
>>
>>
>> Thanks,
>>
>> Josh
>>
>>
>> On Wed, May 30, 2012 at 8:38 AM, Thomas Herault
>>
>> <herault.thomas at gmail.com> wrote:
>>
>> Le 30 mai 2012 a 01:44, George Bosilca a écrit:
>>
>>
>> The ticket has been voted down. Come back in 6 months, maybe 3.1. The
>> votes were 7 yes, 4 abstains and 5 no.
>>
>>
>> Thomas
>>
>>
>> Le 30 mai 2012 à 07:02, Josh Hursey a écrit :
>>
>>
>> How did the vote go for the fault tolerance ticket 323?
>>
>>
>> -- Josh
>>
>>
>> --
>>
>> Joshua Hursey
>>
>> Postdoctoral Research Associate
>>
>> Oak Ridge National Laboratory
>>
>> http://users.nccs.gov/~jjhursey
>>
>> _______________________________________________
>>
>> mpi3-ft mailing list
>>
>> mpi3-ft at lists.mpi-forum.org
>>
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>>
>>
>>
>> _______________________________________________
>>
>> mpi3-ft mailing list
>>
>> mpi3-ft at lists.mpi-forum.org
>>
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>>
>>
>>
>>
>> --
>>
>> Joshua Hursey
>>
>> Postdoctoral Research Associate
>>
>> Oak Ridge National Laboratory
>>
>> http://users.nccs.gov/~jjhursey
>>
>>
>> _______________________________________________
>>
>> mpi3-ft mailing list
>>
>> mpi3-ft at lists.mpi-forum.org
>>
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>>
>>
>> --
>>
>> * Dr. Aurélien Bouteiller
>>
>> * Researcher at Innovative Computing Laboratory
>>
>> * University of Tennessee
>>
>> * 1122 Volunteer Boulevard, suite 350
>>
>> * Knoxville, TN 37996
>>
>> * 865 974 9375
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>>
>> mpi3-ft mailing list
>>
>> mpi3-ft at lists.mpi-forum.org
>>
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>>
>>
>> _______________________________________________
>>
>> mpi3-ft mailing list
>>
>> mpi3-ft at lists.mpi-forum.org
>>
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>>
>>
>>
>> _______________________________________________
>>
>> mpi3-ft mailing list
>>
>> mpi3-ft at lists.mpi-forum.org
>>
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>>
>>
>>
>> _______________________________________________
>> mpi3-ft mailing list
>> mpi3-ft at lists.mpi-forum.org
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>>
>>
>> ________________________________________________________________________
>> Martin Schulz, schulzm at llnl.gov, http://people.llnl.gov/schulzm
>> CASC @ Lawrence Livermore National Laboratory, Livermore, USA
>>
>>
>>
>> _______________________________________________
>> mpi3-ft mailing list
>> mpi3-ft at lists.mpi-forum.org
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>>
>>
>>
>> _______________________________________________
>> mpi3-ft mailing list
>> mpi3-ft at lists.mpi-forum.org
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>
>
>
> _______________________________________________
> mpi3-ft mailing list
> mpi3-ft at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

-- 
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://users.nccs.gov/~jjhursey