[MPI3 Fortran] ASYNC attribute

Wed May 6 18:55:27 CDT 2009

Your questions may already have been answered, I'm late getting to my  
emails.

On May 6, 2009, at 8:42 AM, Rolf Rabenseifner wrote:

> Craig,
>
> Looks good.
> I've a few questions:
> - What is the difference between ASYNC_EXTERNAL and ASYNC?

I made a mistake, there is only one attribute name and the actual name  
is subject to change (perhaps ASYNC_PENDING).

>
> - What are the rules of handing over
>  1. an ASYNC_EXTERNAL real variable or ASYNC dummy argument
>     handed over to a subroutine with a dummy argument
>     defined ___with__ ASYNC

Copyin/copyout is not allowed and code motion is disallowed around any  
call to procedures other than intrinsics.

>
>  2. same but
>     defined _without_ ASYNC

Both the actual and the dummy must have the attribute.  The attribute  
on the actual may be limited to the scope that initiates the call  
(perhaps up the call tree).  It doesn't have to apply everywhere.   
However explicit interfaces must be provided for this entire call chain.

>
>  3. a __normal__ variable or normal dummy argument handed over
>     to a subroutine with a dummy argument defined ___with__ ASYNC

You have to give the __normal__ actual argument the attribute in the  
scope of the call.  The scope may be limited with a block construct.   
Within procedure (or block) scope:

ASYNC :: buf

is sufficient

>
>
>  I would expect that all three cases should be allowed.
>
> I would like to have the following detailed behavior:
> - Copyin/out is prohibited in all 3 cases, i.e.,
>  if MPI_Irecv declares buf as ASYNC then it should prohibit
>  copyin/out when MPI_Irecv is called, and also if buf is
>  declared outside as ASYNC_EXTERNAL then also coyin/out
>  should be prohibited when buf is used as actual argument
>  in a function or subroutine call.
>
> - The scope of ASYNC or ASYNC_EXTERNAL is not inherited by
>  called routines. The called routines must set ASYNC on dummy
>  arguments.

We just took out the "inheritance".  It MUST be explicitly provided  
within the scope of the call routine.

>
>
> - As long as ASYNC or ASYNC_EXTERNAL is valid within the scope
>  of a block, all accesses to that variable must not be moved
>  around any subroutine or function call.

Yes.

>
>
> - The fortran compiler is allowed to modify ASYNC or ASYNC_EXTERNAL
>  variables or parts of them only if this is coded by the application.

Not sure what you mean but I think the answer is yes.  You get what  
you expect.

>
>
> - For accesses between two function or subroutine calls,
>  the compiler can cache ASYNC or ASYNC_EXTERNAL variables in
>  registers to optimize operations or repeated accesses.
>  (This should be a significant difference to VOLATILE!!!)

Yes, definitely not VOLATILE!  Like ASYNCHRONOUS attribute for I/O.

>
>
> Reason for my wishes:
> - Each function or subroutine call can act as a synchronization point
>  where the application is informed that some data is stored or used
>  by asynchronously routing routines.

I think I need an example.

>
>
> Implications:
> - In your example, one needs
>   SUBROUTINE UPDATE_INTERIOR(buf)
>   TYPE(*), DIMENSION(..), ASYNC   :: buf
>  This is needed to avoid, e.g., temporary aching, overwriting
>  and restoring of parts of the data for loop fusion.

Yes (essentially copyin/copyout is prohibited).

>
>
> - In your example, buf cannot be transfered with copyin/out
>  in the call to UPDATE_BOUNDARIES, because "ASYNC_EXTERNAL :: buf"
>  is already defined in the calling block.

I think so.  I'll have to ask about this if UPDATE_BOUNDARIES doesn't  
have the ASYNC attribute.

>
>
> Open question:
> - Do we want to have a method, to tell that a call acts as a
>  synchronization for one or several buffers?

Not needed.

>
>  And that other calls do not act as synchronization point.
>  I.e., do we want a method to change the default:
>    Default is, that any routine can act as synchronization point,
>    i.e., any (register) caching must be done between such points.
>    This means any BLAS or LAPACK routine is handled as a call
>    to MPI_WAIT.

If I understand what you mean, synchronization points (MPI_Wait?) are  
not special.  Code motion is inhibited regarding ALL routines.  I'll  
have to ask about register usage.  Certainly ASYNC variables can be  
stored in registers.  I don't think registers are valid across any  
procedure call.

>
> - I see two possibilities:
>  A) To change this default by some special syntax
>  B) Not to use this default at all.
>     Proposal:
>     Synchronizing routines must be declared as "SYNC_EXTERNAL"

We discussed this and there was no need for synchronizing routines to  
be marked as special.

>
>
>     I.e.,
>     SUBROUTINE MPI_Wait(request, status, err) SYNC_EXTERNAL, BIND(C,  
> name="MPI_Wait")
>
>     This means, that the portions of a variable, that are written
>     asynchronously and the portions that are written localy
>     of a ASYNC or ASYNC_EXTERNAL variable can be changed at
>     any call where the variable is an actual argument
>     or where the called routine has the SYNC_EXTERNAL flag.

Yes.  However, the user must to the correct thing so that a correct  
program is produced.  The programmer can't write to the same memory  
location that MPI is.

>
>
>     In the MPI API, we would give all WAIT, TEST and communication
>     routines this SYNC_EXTERNAL flag.
>     Reason:
>      -- WAIT and TEST: obvious.
>      -- communication routines can have invisible buffer arguments
>         due to the use of absolut addresses in MPI derived
>         datatypes and usage of MPI_BOTTOM

Again, the attribute goes on the variable, so the compiler will do the  
correct thing within the scope of the variable.

>
>
> What do you think about this behavior?

I think we've got everything we need.

>
>
> Best regards
> Rolf
>
> On Tue, 5 May 2009 17:45:00 -0600
> Craig Rasmussen <crasmussen at newmexicoconsortium.org> wrote:
>> I'm at the WG5 Fortran standards meeting and I think we've made  
>> significant advances regarding the MPI-3 Fortran API.  We are  
>> discussing adding a new variable attribute, ASYNC.  I'd like to get  
>> feedback from the MPI-3 Fortran working group on these possible  
>> changes.
>> ASYNC implies that a variable is potentially active in asynchronous  
>> operations outside of Fortran.  The use of this new attribute  
>> should give the compiler enough information to inhibit  
>> optimizations similar  to inhibitions involved in Fortran  
>> asynchronous I/O, specifically  copyin/copyout and code motion.   
>> The compiler will likely have to  inhibit code motion regarding any  
>> use of a variable with the ASYNC  attribute and procedures other  
>> than intrinsics.  The affect is similar  to the use of the volatile  
>> attribute (but not regarding loads and  stores).
>> Usage is outlined below:
>> ----------------
>> real, ASYNC_EXTERNAL :: buf(100,100)
>> type(MPI_Request) :: request
>> ! initiate data transfer of boundary
>> CALL MPI_IRecv(buf(:,1),...,request,...)  ! no copyin/copyout will  
>> happen
>> ! do work on interior of the buffer while transfer is in progress
>> CALL UPDATE_INTERIOR(buf)  ! dummy arg should have ASYNC attribute
>> ! wait for the communication to finish
>> CALL MPI_Wait(request,...)   ! no code motion allowed with respect  
>> to  buf
>> ! finish work on buffer boundaries
>> CALL UPDATE_BOUNDARIES(buf)  ! should copyin/copyout be allowed here?
>> ----------------
>> So how does this look?  Anything we've left out?  I'm also  
>> including the interface definition.
>> - craig
>> ----------------
>> interface
>> SUBROUTINE MPI_Irecv(buf, count, datatype, source, tag, comm,  
>> request,  err) BIND(C, name="MPI_Irecv")
>>   import MPI_Datatype, MPI_Comm, MPI_Request
>>   TYPE(*), DIMENSION(..), ASYNC   :: buf
>>   integer, value,     intent(in)  :: count
>>   type(MPI_Datatype), intent(in)  :: datatype
>>   integer, value,     intent(in)  :: source
>>   integer, value,     intent(in)  :: tag
>>   type(MPI_Comm),     intent(in)  :: comm
>>   type(MPI_Request),  intent(out) :: request
>>   integer,            intent(out) :: err
>> END SUBROUTINE MPI_Irecv
>> SUBROUTINE MPI_Isend(buf, count, datatype, dest, tag, comm,  
>> request, err) BIND(C, name="MPI_Isend")
>>   import MPI_Datatype, MPI_Comm, MPI_Request
>>   TYPE(*), DIMENSION(..), ASYNC   :: buf
>>   integer, value,     intent(in)  :: count
>>   type(MPI_Datatype), intent(in)  :: datatype
>>   integer, value,     intent(in)  :: dest
>>   integer, value,     intent(in)  :: tag
>>   type(MPI_Comm),     intent(in)  :: comm
>>   type(MPI_Request),  intent(out) :: request
>>   integer,            intent(out) :: err
>> END SUBROUTINE MPI_Isend
>> SUBROUTINE MPI_Wait(request, status, err) BIND(C, name="MPI_Wait")
>>   import MPI_Request, MPI_Status
>>   type(MPI_Request), intent(in)  :: request  !	inout?
>>   type(MPI_Status),  intent(out) :: status
>>   integer,           intent(out) :: err
>> END SUBROUTINE MPI_Wait
>> end interface
>> _______________________________________________
>> mpi3-fortran mailing list
>> mpi3-fortran at lists.mpi-forum.org
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran
>
> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
> Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)