[Mpi3-ft] Communicator Virtualization as a step forward

Thu Feb 19 02:43:44 CST 2009

Dear Thomas,

Thank you. I do not propose a certain checkpointer. I just assume that the MPI and the checkpointer involved will find each other somehow (this may be a configuration option, dynamic detection, whatever), and then they will arrange for the proper interaction. The proposed calls are just marking the point in the program where the MPI is prepared for checkpointing (MPI_PREPARE_FOR_CHECKPOINT) and is ready to resume after checkpoint (MPI_RESTORE_AFTER_CHECKPOINT).

Depending on what the MPI and the checkpointer agree to do, the actual scope of the data saved will vary. What is proposed, in essence, is a couple of brackets that act almost like MPI_Finalize (MPI_PREPARE_FOR_CHECKPOINT) and MPI_Init (MPI_RESTORE_AFTER_CHECKPOINT), with the reservations wrt connection re-establishment, data types, etc. that you so appropriately pointed out.

Note that the MPI_PREPARE_FOR_CHECKPOINT may not need to do any disconnects or remove or relocate any MPI objects. This depends on the checkpointer involved and the agreement between it and the MPI. Likewise, the MPI_RESTORE_AFTER_CHECKPOINT may not need to restore any objects if there was not restart in between. This needs to be communicated to the MPI.

One way would be thru the callbacks that would be called from within the MPI_PREPARE_FOR_CHECKPOINT to notify the checkpointer that the MPI is ready, and another callback, or the same callback with another argument (e.g., 0 for preparation, 1 for restoration) that should tell the MPI whether restart actually took place. This can be done thru the callback return code or something. These callbacks can be registered with the MPI thru, say, MPI_REGISTER_CHECKPOINT_CALLBACK or something.

Another way would be to extend the closing brackets into two (MPI_CONTINUE_AFTER_CHECKPOINT for simple continuation, MPI_RESTORE_AFTER_RESTART for actual restart), and a query function to ask the checkpointer at hand whether restart happened. In that case the aforementioned callback would only notify the checkpointer of the respective events.

Note also that what the checkpointer actually does is up to the checkpointer. So, this may be a WRITE statement, if this is all the application really needs.

A lot of flesh is yet to be added to these bones (scenarios, synchronization, callbacks, etc.), but I think that the concept is basically sound.

Best regards.

Alexander

-----Original Message-----
From: mpi3-ft-bounces at lists.mpi-forum.org [mailto:mpi3-ft-bounces at lists.mpi-forum.org] On Behalf Of Thomas Herault
Sent: Wednesday, February 18, 2009 7:53 PM
To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group
Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward

Le 18 févr. 09 à 13:34, Supalov, Alexander a écrit :

> Thanks. Let me address your objections one by one below.
>
> -----Original Message-----
> From: mpi3-ft-bounces at lists.mpi-forum.org [mailto:mpi3-ft-bounces at lists.mpi-forum.org
> ] On Behalf Of Greg Bronevetsky
> Sent: Wednesday, February 18, 2009 6:43 PM
> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working
> Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working
> Group
> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward
>
> First, this API works only for the synch-and-stop protocol. It does
> not support any other protocol and this will lead into bad places in
> the future because this protocol scales poorly with increasing
> failure rates since it requires all processes to participate in every
> single recovery. I can discuss the reasons for my assertion in more
> detail if you'd like.
>
> AS> What other protocols are you talking about? I did not tell
> anywhere whether these calls synch anything, or whether they are
> collective, or whether the checkpointer is called on all nodes. This
> remains to be defined. If you would please define other scenarios,
> including those that scale well, I bet we'd probably be able to map
> them to these calls that way or another.
>
> Second, this API is not compatible with application-level
> checkpointing. An application-level checkpointer can deal with
> internal MPI state in one of two ways. One option is to checkpoint it
> at a high-level by using only the existing MPI calls, performing the
> appropriate tracking of messages at calls to MPI_Send and MPI_Recv.
> The simplest example of this is to ensure that there is no
> communication going on at the time of the checkpoint, and then save
> each process' state immediately after calling MPI_Barrier(). If this
> is the option used, there is no need for any API extensions.
>
> AS> One option for the application level checkpointing is to save
> the apps specific state (those two vectors) without saving the state
> of the MPI at all. This will work if the MPI state is quiet (no
> connections, nothing), and the application can be sure that its
> state will be good, and when it restarts, the MPI hitting the
> MPI_restore_after_checkpoint will know how to re-establish all
> connections, etc. I think this is more than an MPI_Barrier, isn't it?
>

Hello Alexander, Greg,

one point that was raised in the last forum meeting is that part of
the application state is related to the MPI library internal state:
for example, does the application expect that the communicators it
created still exist after the restart? What about datatypes, etc...?

Restarting MPI library is more than just re-establishing the
connections. And that is Greg's point:
  - either we believe that the checkpointing system will save
something for the MPI library (like the internal representation of the
communicators, and which handles are connected to them), and then what
the checkpointer will save for the MPI library must be specified
somewhere

  - or the checkpointing system saves nothing for the MPI library.
Here we have two new approaches:
    - either the user cannot expect anything except what is defined
after MPI_Init to be still working. This approach advocates a simple
modification of the standard: allowing an application to call
MPI_Finalize(); ... MPI_Init(); multiple times during its execution.
It could checkpoint between MPI_Finalize and MPI_Init

     - or design some way for the MPI library to save its internal
state somehow. This could be done, but many system-level checkpointing
systems do not provide functionalities to exclude the MPI library from
the checkpoint, so the question arises on what can a MPI library do to
cope with a partial initialization (restored by an unknown system-
level checkpointer), and a state to restore, saved somewhere?

> The second option is for the application to somehow same MPI internal
> data structures, which is what the MPI_Prepare_for_checkpoint call is
> for: to make sure that all MPI state is in locations accessible by
> the application. Where is that? Well, it depends on the operating
> system and the hardware. Does the application need to checkpoint and
> restore MPI's open file handles or shared memory regions or is MPI
> supposed to let those go during MPI_Prepare_for_checkpoint? The point
> is, all these details cannot be defined in the MPI spec and if this
> is the case, the call will not be sufficiently well-defined to be
> used by an application-level checkpointer. However, it can be used by
> the application to initiate a system-level checkpoint since in this
> case all those details will not be the application's problem.
>
> AS> As for the place where all relevant MPI data is to be stored at
> this point in time, we should not care - the MPI implementation and
> the checkpointer involved will agree on this when the MPI will be
> configured to use this particular checkpointer, either statically or
> dynamically. The checkpointer will know that the data is in the
> "right" place when it gats control after the
> MPI_prepare_for_checkpoint call.
>

-> So, in your solution, an MPI library will come with a specific
checkpointing tool? Do you enforce all implementations to come with a
checkpointing tool? Checkpointing tools (system-level at least) are OS
and hardware dependent. This is an issue for portability. And one
could argue that it is not the role of the MPI library to save the
application state. Not in the approach we decided to take for fault
tolerance in MPI-3.

> Finally, it does support system-initiated checkpointing. If the
> system wants to start a checkpoint in the middle of an MPI_Send()
> call, this API provides no help. If we care about this use-case we
> need to allow checkpoints to be taken at all times or provide a way
> to register a callback that tells the system when a checkpoint can
> next be taken.
>
> AS> I suppose this is "does not support", right? If so, let's simply
> add this callback registration call, say,
> MPI_REGISTER_CHECKPOINT_CALLBACK, that will define the call to be
> made when the MPI_Prepare_for_checkpoint is about to return control
> to the application. This will I guess address your concern, won't it?
>
> The bottom line given the above issues is that the only benefit this
> API does have is providing a common set of names for functions that
> have poor semantics and limited functionality. This benefit does not
> pass the bar.
>
> AS> See above. I think you raise valid points, but all of them can
> be adequately addressed given the desire to provide the checkpoint-
> restart capability
>

I agree with you. But I ask how far away from the initial goal we are
willing to go, in order to provide checkpoint/restart capability?

If the MPI library must embed (or be configured for) a checkpointing
mechanism, we do not need to expose any API, or modify the existing
standard: this has been done by many MPI implementations for multiple
specific checkpointers. In this case, you do not need to expose
anything to the application and do everything internally in the library.

> If I'm allowed to digress here and refer back to the fault
> notification, note that the checkpoint/restart capability is
> something that does guarantee fault tolerance, also in the case when
> fatal unrecoverable errors occur. You "simply" fix the machine and
> restart from the earlier checkpoint.
>
> As such, this widely used and proven capability may be a safe first
> bet for the FT WG to introduce as something people will like and
> will use. Much more likely, in fact, than the notification stuff
> that yet needs to be properly defined, and as currently envisioned,
> with all that uncertain communicator restoration stuff, will require
> substantial application rethinking and repgrogramming to be used.
>
> After all, this is only a matter of priorities. Going for a low
> hanging fruit like checkpoint/restart may be unromantic, but it's
> definitely very pragmatic and sound. And in any case, I call upon
> the same set of criteria to be used for all things that we consider.
> Usability in the sense that a feature will be used by commercial
> apps is definitely one of them. Or so I think.
>
> Greg Bronevetsky
> Post-Doctoral Researcher
> 1028 Building 451
> Lawrence Livermore National Lab
> (925) 424-5756
> bronevetsky1 at llnl.gov
> http://greg.bronevetsky.com
>

Thomas

> At 09:27 AM 2/18/2009, Supalov, Alexander wrote:
>> I see. Let me explain what I think those two calls would do to
>> facilitate checkpointing.
>>
>> The program piece would look as follows:
>>
>> CALL MPI_PREPARE_FOR_CHECKPOINT
>>
>> ! Whatever calls are necessary to do the actual checkpointing,
>> including, in the case of resident monitoring checkpointer, none.
>>
>> CALL MPI_RESTORE_AFTER_CHECKPOINT
>>
>> The first call will bring MPI into a state that is acceptable to the
>> checkpointer involved. This may include quiescence, etc.
>>
>> Then the checkpointer will do its job. Note that this may be
>> application level checkpointer that saves two important arrays. Or
>> this may be a system checkpointer that dumps the memory, etc.
>> Finally, this can be a checkpointer that is notified when the
>> MPI_PREPARE_FOR_CHECKPOINT is about to leave and does the job in
>> some way without the program calling it.
>>
>> The second call will restore the MPI state to what it was,
>> functionally, after the checkpoint. "Functionally" here means that
>> this may include running on a different set of nodes, etc.
>>
>> Is it this scenario that you think does not pass the bar? If so, in
>> what way?
>>
>> -----Original Message-----
>> From: mpi3-ft-bounces at lists.mpi-forum.org
>> [mailto:mpi3-ft-bounces at lists.mpi-forum.org] On Behalf Of Greg
>> Bronevetsky
>> Sent: Wednesday, February 18, 2009 6:20 PM
>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working
>> Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working
>> Group
>> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward
>>
>> My point is that each MPI implementation will have to negotiate with
>> each checkpointer what this call will mean on each platform, meaning
>> that it is upto the implementation-specific call to do all that
>> magic. It doesn't save the application any real effort to call this
>> routine by its implementation-specific name rather than a unified
>> name given that the other two major problems are not solved:
>> - Providing a uniform way for checkpointers and MPI implementations
>> to interact (THE major for having this proposal in the first place)
>> and
>> - Support for more than the synch-and-stop checkpointing protocol,
>> which does scales poorly with rising failure rates.
>>
>> My point here is that covering one of the three major issues is not
>> good enough to modify the MPI specification and I don't think that
>> we'll ever be able to do more than that. We're pretty much agreed in
>> the working group that the first point above cannot be done and I
>> don't think that the second can be done without providing callbacks
>> to be called for each incoming/outgoing message.
>>
>> Greg Bronevetsky
>> Post-Doctoral Researcher
>> 1028 Building 451
>> Lawrence Livermore National Lab
>> (925) 424-5756
>> bronevetsky1 at llnl.gov
>> http:// greg.bronevetsky.com
>>
>> At 08:40 AM 2/18/2009, Supalov, Alexander wrote:
>>> Thanks. How will the checkpointer-specific call talk to the MPI?
>>>
>>> -----Original Message-----
>>> From: mpi3-ft-bounces at lists.mpi-forum.org
>>> [mailto:mpi3-ft-bounces at lists.mpi-forum.org] On Behalf Of Greg
>>> Bronevetsky
>>> Sent: Wednesday, February 18, 2009 5:29 PM
>>> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working
>>> Group; MPI 3.0 Fault Tolerance and Dynamic Process Control working
>>> Group
>>> Subject: Re: [Mpi3-ft] Communicator Virtualization as a step forward
>>>
>>>
>>>> Thanks. How will you let the MPI know the checkpoint is coming, to
>>>> give it a fair chance to prepare to this and then recover after the
>>>> checkpoint? This is akin to the MPI_Finalize/MPI_Init in some
>>>> sense,
>>>> midway thru the job, hence the analogy.
>>>
>>> Just use the checkpointer-specific call. The call is going to have
>>> checkpointer-specific semantics, so why not give it a
>>> checkpointer-specific name? I understand that there is some use to
>>> allowing applications to use the same name across all checkpointers
>>> but the bar should be higher than that for adding something to the
>>> standard. Also, right now the whole approach inherently only
>>> supports
>>> one checkpointing protocol: synch-and-stop. If we can work out a
>>> more
>>> generic API that supports other protocols I think that it may have
>>> enough value to be included in the spec. Right now it still hasn't
>>> passed the bar.
>>>
>>> Greg Bronevetsky
>>> Post-Doctoral Researcher
>>> 1028 Building 451
>>> Lawrence Livermore National Lab
>>> (925) 424-5756
>>> bronevetsky1 at llnl.gov
>>> http://  greg.bronevetsky.com
>>>
>>> _______________________________________________
>>> mpi3-ft mailing list
>>> mpi3-ft at lists.mpi-forum.org
>>> http://  lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>>> ---------------------------------------------------------------------
>>> Intel GmbH
>>> Dornacher Strasse 1
>>> 85622 Feldkirchen/Muenchen Germany
>>> Sitz der Gesellschaft: Feldkirchen bei Muenchen
>>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer
>>> Registergericht: Muenchen HRB 47456 Ust.-IdNr.
>>> VAT Registration No.: DE129385895
>>> Citibank Frankfurt (BLZ 502 109 00) 600119052
>>>
>>> This e-mail and any attachments may contain confidential material
>>> for
>>> the sole use of the intended recipient(s). Any review or
>>> distribution
>>> by others is strictly prohibited. If you are not the intended
>>> recipient, please contact the sender and delete all copies.
>>>
>>>
>>> _______________________________________________
>>> mpi3-ft mailing list
>>> mpi3-ft at lists.mpi-forum.org
>>> http://  lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>>
>> _______________________________________________
>> mpi3-ft mailing list
>> mpi3-ft at lists.mpi-forum.org
>> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>> ---------------------------------------------------------------------
>> Intel GmbH
>> Dornacher Strasse 1
>> 85622 Feldkirchen/Muenchen Germany
>> Sitz der Gesellschaft: Feldkirchen bei Muenchen
>> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer
>> Registergericht: Muenchen HRB 47456 Ust.-IdNr.
>> VAT Registration No.: DE129385895
>> Citibank Frankfurt (BLZ 502 109 00) 600119052
>>
>> This e-mail and any attachments may contain confidential material for
>> the sole use of the intended recipient(s). Any review or distribution
>> by others is strictly prohibited. If you are not the intended
>> recipient, please contact the sender and delete all copies.
>>
>>
>> _______________________________________________
>> mpi3-ft mailing list
>> mpi3-ft at lists.mpi-forum.org
>> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>
> _______________________________________________
> mpi3-ft mailing list
> mpi3-ft at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
> ---------------------------------------------------------------------
> Intel GmbH
> Dornacher Strasse 1
> 85622 Feldkirchen/Muenchen Germany
> Sitz der Gesellschaft: Feldkirchen bei Muenchen
> Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer
> Registergericht: Muenchen HRB 47456 Ust.-IdNr.
> VAT Registration No.: DE129385895
> Citibank Frankfurt (BLZ 502 109 00) 600119052
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
>
>
> _______________________________________________
> mpi3-ft mailing list
> mpi3-ft at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
>

_______________________________________________
mpi3-ft mailing list
mpi3-ft at lists.mpi-forum.org
http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft
---------------------------------------------------------------------
Intel GmbH
Dornacher Strasse 1
85622 Feldkirchen/Muenchen Germany
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer
Registergericht: Muenchen HRB 47456 Ust.-IdNr.
VAT Registration No.: DE129385895
Citibank Frankfurt (BLZ 502 109 00) 600119052

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.