[Mpi-22] Proposed amendment to application cleanup callback text

[Quincey Koziol]

Hi all,
        I've got a problem which I think might be a candidate for a fix in  
the 2.2 standard:  Is it legal to make MPI calls in the 'delete'  
callback for an attribute attached to MPI_COMM_SELF which is being  
freed during a call to MPI_Finalize?

        Section 4.8 in the 2.0 standard doesn't really say, I think.  Here's  
the text:

> 4.8 Allowing User Functions at Process Termination
>
> There are times in which it would be convenient to have actions  
> happen when an MPI process
> finishes. For example, a routine may do initializations that are  
> useful until the MPI job (or
> that part of the job that being terminated in the case of  
> dynamically created processes) is
> finished. This can be accomplished in MPI-2 by attaching an  
> attribute to MPI_COMM_SELF
> with a callback function. When MPI_FINALIZE is called, it will first  
> execute the equivalent
> of an MPI_COMM_FREE on MPI_COMM_SELF. This will cause the delete  
> callback function
> to be executed on all keys associated with MPI_COMM_SELF, in an  
> arbitrary order. If no
> key has been attached to MPI_COMM_SELF, then no callback is invoked.  
> The “freeing” of
> MPI_COMM_SELF occurs before any other parts of MPI are affected.  
> Thus, for example,
> calling MPI_FINALIZED will return false in any of these callback  
> functions. Once done with
> MPI_COMM_SELF, the order and rest of the actions taken by  
> MPI_FINALIZE is not specified.
>
> 	Advice to implementors. Since attributes can be added from any  
> supported language,
> 	the MPI implementation needs to remember the creating language so  
> the correct
> 	callback is made. (End of advice to implementors.)

        Here's my use case: we cache modified file information in the HDF5  
library and need to be able to flush that data to the file if the user  
calls MPI_Finalize without calling the HDF5 routines to close (or  
flush) that file first.  It looks like adding an attribute onto  
MPI_COMM_SELF is the right thing to do, and the standard seems to  
imply that it's possible to make MPI calls from that callback, but  
it's not certain to me.  (Obviously, the HDF5 library needs to make I/ 
O calls and possible may need to exchange data or invoke a barrier, etc)

        Is it reasonable to add a sentence to the effect that "all MPI  
operations (except MPI_Finalize and MPI_Init) are possible from the  
attribute's 'delete' callback"?

        Here's a copy of section 4.8, with text to that effect worked in:

> 4.8 Allowing User Functions at Process Termination
>
> There are times in which it would be convenient to have actions  
> happen when an MPI process
> finishes. For example, a routine may do initializations that are  
> useful until the MPI job (or
> that part of the job that being terminated in the case of  
> dynamically created processes) is
> finished. This can be accomplished in MPI-2 by attaching an  
> attribute to MPI_COMM_SELF
> with a callback function. When MPI_FINALIZE is called, it will first  
> execute the equivalent
> of an MPI_COMM_FREE on MPI_COMM_SELF. This will cause the delete  
> callback function
> to be executed on all keys associated with MPI_COMM_SELF, in an  
> arbitrary order. If no
> key has been attached to MPI_COMM_SELF, then no callback is invoked.  
> The “freeing” of
> MPI_COMM_SELF occurs before any other parts of MPI are affected and  
> all MPI operations (except MPI_Finalize and MPI_Init) are available  
> to the attribute's 'delete' callback. Thus, for example,
> calling MPI_FINALIZED will return false in any of these callback  
> functions. Once done with
> MPI_COMM_SELF, the order and rest of the actions taken by  
> MPI_FINALIZE is not specified.
>
> 	Advice to implementors. Since attributes can be added from any  
> supported language,
> 	the MPI implementation needs to remember the creating language so  
> the correct
> 	callback is made. (End of advice to implementors.)

        Thanks,
                Quincey Koziol
                The HDF Group