[MPI3 Fortran] MPI buffer arguments - void and call-by-reference

[Craig Rasmussen]

After the J3 meeting in Las Vegas last week I think we are reaching a  
conclusion on many of the MPI 3 interface issues.  The one remaining  
point of disagreement is whether to use the ASYNCHRONOUS attribute  
(currently it only applies to Fortran IO) or the VOLATILE attribute.   
Since there is so much disagreement on this issue within J3 and WG5  
(this includes disagreements between several influential members on  
the committees and some even switching sides from time to time), I'll  
leave the solution for later.  As for now, I believe that the MPI  
Forum should disregard the J3 paper I send to you earlier, supposedly  
clarifying the asynchronous MPI question.

We do have clarity on another issue, however.  There was substantial  
agreement on the committee for the following interface (I was the lone  
dissenting vote):

interface
     subroutine MPI_Send(buf, ...) BIND(C)
         TYPE(*), DIMENSION(..), INTENT(in) :: buf
         ...
     end subroutine
end interface

This means that actual argument associated with the dummy argument  
buf, can be an array or scalar of any type.  If it is an array, it can  
have any rank.  The major impact to the MPI implementation is that the  
buf argument will be passed by descriptor.  This means that there will  
be no copyin/copyout of the array by the Fortran compiler.  The MPI  
implementation is free to enforce whatever semantics it wishes on the  
buf argument.  In your example below,

    CALL MPI_SEND(rarr(5:30:3),...)

the MPI implementation can get the address of the rarr array and the  
5:30:3 index information.  It can choose to return an error if it  
wishes to enforce that the array is contiguous or it can ignore the  
stride by 3 information and just proceed as if the array was contiguous.

I'm pretty sure this will be the only choice provided in Fortran 2008/ 
Interop TR  and MPI 3.0 timeframe.  Otherwise the MPI Forum can use  
the CLOC() intrinsic in the current 2003 interoperability standard to  
pass void* buffer addresses.

I think using TYPE(*) and DIMENSION(..) is an excellent solution for  
MPI 3.0.  The advantages are:

1. Current user code won't have to change beyond using the MPI3 module.
2. The interface is non-generic and doesn't have the interface  
explosion in the current MPI 90 interfaces.  This interface will even  
accept scalar actuals.
3. The interface is type safe as far as the other (non void*) arguments.
4. The actual argument associated with buf doesn't require the TARGET  
attribute (as the CLOC/C_PTR route does), with the possibly  
substantial performance implications.

Potential downside:

1. There must be a wrapper that calls the MPI C implementation.  We  
had decided on this anyway as we wanted to stick with the SUBROUTINE  
syntax.
2. There will be some minor performance implications as the array  
descriptor must be examined before calling the MPI C implementation.   
The performance hit can be as minor as dereferencing a struct to get  
at the array address to something more substantial if the  
implementation wants to make use of the stride information.

I'm hoping we can look at this in more detail at the next MPI Forum  
meeting and reach agreement on the non-asynchronous interfaces.

We can also discuss MPI_Isend and MPI_Irecv and what to do with them.   
I'm assuming that the use of ASYNCHRONOUS plus TARGET will be  
preferred over VOLATILE.  Since at this point ASYNCHRONOUS really only  
pertains to Fortran IO, MPI Forum could request another name for an  
equivalent attribute, perhaps one that combines the semantics of  
ASYNCHRONOUS and TARGET.

Cheers,
Craig


On Feb 9, 2009, at 3:59 PM, Rolf Rabenseifner wrote:

> This is the second part of my two mails:
>
> B) VOID buffer declaration
> --------------------------
>
> B.1 Current status
>
>    INTEGER :: ibuf
>    CALL MPI_SEND(ibuf,...)
>
>    REAL, DIMENSION(100) :: rarr
>    CALL MPI_SEND(rarr,...)
>    CALL MPI_SEND(rarr(5:30:3),...)
>
>    COMPLEX, DIMENSION(10,20,30) :: carr3dim
>    CALL MPI_SEND(carr3dim,...)
>
>    TYPE buff_type
>     SEQUENCE
>     INTEGER, DIMENSION(3):: i_val
>     DOUBLE PRECISION, DIMENSION(5):: d_val
>    END TYPE buff_type
>    TYPE (buff_type) :: struct
>    CALL MPI_SEND(struct,...)
>
>    TYPE (buff_type), DIMENSION(10,20) :: struct2dim
>    CALL MPI_SEND(struct2dim,...)
>
>    ... ... ... ...
>
>    All this works and the C-writte MPI_SEND receives the address
>    of the first data element of each buffer.
>
>    Currently, there isn't a F90 interface definition
>    (i.e. only F77 style is used)
>
>    MPI has its own interface to handle the data structure behind
>    the starting pointer, the so called MPI datatypes.
>    Such an MPI datatype  need not to describe the whole
>    structure, i.e., there may be differences between the
>    type information that was told the compiler and the MPI datatype
>    information.
>
>
> B.2 Goals
>
>    B.2.1 To allow F90 interfaces that do not break eisting codes
>    B.2.2 To minimize the burden for MPI libraries implementers
>          and Fortran compiler implementers.
>
> B.3 Maybe a simple but efficient solution
>
>    SUBROUTINE MPI_SEND(buf,...
>    VOID :: buf
>    END
>
>    This can be used only in a interface definition for routines
>    that are implemented in C.
>    There is no meaning if one wants to write the routine MPI_SEND
>    in Fortran.
>
>    Meaning of VOID buf:
>    Exactly the same as if implicit (F77) interfaces are used.
>
>    (Not needed for MPI, but a possible extension:
>
>       Instead of VOID, one is using IMPLARG, meaning the
>       same as VOID on the caller side and is also allowed
>       on the soubroutine side.
>       Meaning: Same argument handlich as with implicit interfaces,
>                i.e., handing over the pointer to the data or
>                in cases of noncontiguos subarrays only a pointer to
>                a contiguous copy of the data.
>    )
>
>
> C) Call by reference and not by in-copy-out-copy
> ------------------------------------------------
>
>   It is clear, that F77 interfaces cannot do strided arrays
>   directly with call-by-reference.
>
>   For contiguous arrays and non-arrays, it isn't a problem.
>
>   Fortran compilers are allowed to choose call-by-incopy-and-outcopy
>   instead of call-by-reference.
>
>   Maybe the following additional option may solve the problem:
>
>   SUBROUTINE MPI_SEND(buf,...
>   VOID, CALLBYREFERENCE :: buf
>
>   Meaning: Same as what currently done, i.e., call-by-reference
>   on the caller side, if call-by-reference is possible (i.e.,
>   not for non-contiguous subarrays)
>
>
> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
> Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)