[MPI3 Fortran] Fortran extra_state argument to MPI attribute functions

[Jeff Squyres]

A thread started on the general MPI-2.2 list a while ago asking about  
the extra_state parameter to the fortran callbacks in the MPI  
attribute functions:

http://lists.mpi-forum.org/mpi-22/2009/04/0395.php
and continued here
http://lists.mpi-forum.org/mpi-22/2009/05/0407.php

Specifically, there is no indication of how the "extra_state"  
parameter is handled when it is passed to callback functions.

Bear with me -- this requires a little explanation...

The issue is this: when you create a "keyval" in MPI, you specify 2  
callback functions and an INTEGER extra_state parameter.  Each of the  
two callback functions has an extra_state parameter; the intent is  
that the extra_state from the keyval creation function is passed as  
the argument to these two callback functions.  Here's a sample keyval  
creation:

    ! copy_fn and delete_fn are subroutines
    ! comm is an INTEGER
    INTEGER extra_state
    extra_state = 42
    CALL MPI_Comm_create_keyval(copy_fn, delete_fn, comm, extra_state,  
ierr)
    extra_state = 17

There are three options for how this extra_state argument is treated:

1. The extra_state that is passed to the callback functions is the  
exact reference that was passed to MPI_Comm_create_keyval().  Hence,  
when copy_fn/delete_fn are invoked (after MPI_Comm_create_keyval()  
returns), they'll get a passed value of 17.

--> This seems to be obviously illegal in Fortran.  If you pass an  
expression as the value of the extra_state argument to  
MPI_Comm_create_keyval, that reference will be out of scope by the  
time the callbacks are invoked.  Unfortunately, some MPI  
implementations (including Open MPI, until version 1.3.3 is released!)  
use this semantic.  Oops.  :-\

2. Copy the value of extra_state during the call to  
MPI_Comm_create_keyval() into internal MPI implementation storage.   
When the callbacks are invoked, pass a reference to this internal  
storage.  Hence, the first callback that is invoked will always  
receive an extra_state value of 42 (not 17).

Note that no INTENT is specified in the MPI document prototypes for  
the copy_fn and delete_fn, so the callback may actually modify the  
extra_state value.

2a. The MPI implementation can always pass the same extra_state  
reference, meaning that all subsequent callback invocations will get  
the most-recently-modified value.  Hence, if one callback modifies  
extra_state to be 94, then the next callback will receive an  
extra_state value of 94.

2b. The MPI implementation can copy the internal storage and give a  
unique copy to each callback invocation so that *all* invocations will  
always get the original extra_state value (42), even if extra_state is  
modified during the callback.

Now that explanation is over, let me explain the intent of this  
interface from the C side; I believe it gives insight into what should  
happen on the Fortran side.

The C prototype for the callback functions have (void*) extra_state  
arguments.  The intent for them is just like many other C interfaces  
with extra_state arguments: they can be anything, to include pointers  
into much larger / external data instances.  Hence, it's not the  
*value* of the extra_state that is overly important -- that value is  
usually a pointer to something else.

The people who made the MPI interface for the callbacks intended that  
the Fortran extra_state parameter should be used in the same way.   
With Fortran 77, you could just pass my_array(1) as the extra_state  
argument, and then your extra_state argument in the callback would  
effectively be a reference to the whole my_array array.  Ugly, but it  
worked.  :-\  With tighter bindings from F90 and onward, this abuse is  
clearly not possible (which is a Good Thing!).

However, the original intent is somewhat lost here -- since the  
MPI_COMM_CREATE_KEYVAL interface specifies that the EXTRA_STATE  
parameter is an integer, how can you pass a reference to a larger /  
external structure?  This seems somewhat broken / inconsistent with  
the intent from the C interface.

Of course, you *could* pass an INTEGER EXTRA_STATE value that is an  
index into a global array in the application that contains some  
reference to an external data instance -- but that seems fairly  
manual.  Is there a better way?

How are such things normally done in Fortran?  (meaning: what *should*  
we do here?)

As an aside: I think that the above case means that we should go with  
option 2b (meaning: we need to add text to MPI 3.0 indicating that 2b  
semantics must be observed).  Indeed, this would be like most other C  
(void*) extra state interfaces: the value that is passed is *always*  
the same as the original value (42), regardless of whether one of the  
callbacks changed the value locally or not.  The fact that Fortran  
passes parameters by reference and passes by value means that if we  
want the C semantics, we need to use option 2b.

-- 
Jeff Squyres
Cisco Systems