[MPI3 Fortran] Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION

Fri Oct 7 09:48:17 CDT 2011

I guess this looks ok -- there's some grammar changes I'd make, though.

One thing to be careful of:

> * Advice to users.
> * Structures combining different basic datatypes should be defined so that there will
> * be no gaps based on alignment rules, and if used as an array of structures, then
> * also without such an alignment-gap at the end of the structure.
> * In MPI communication, the content of such gaps would not be communicated.

^^^ This is not necessarily true.  A smart enough MPI implementation may realize that it's faster to treat a small gap as a contiguous datatype and just ignore the extra data, especially if it's in a homogeneous environment, and copy-in/copy-out is used anyway.

On Oct 6, 2011, at 7:55 PM, Rolf Rabenseifner wrote:

> Dear all,
> 
> here is my proposal to solve the alignment problems.
> Please check carefully.
> 
> It is touching only the Fortran side.
> C problems with alignments are not really touched, but should be small.
> 
> ----------------------------------------------------------------------
> 
> (Lines with * are new, lines with : are modified, lines with x are removed)
> 
> mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
> p86:45-47 (=MPI-2.2 p78:45-47) read
> 
>  If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon 
>  is the least non-negative increment needed to round extent(Typemap) to the next multiple 
>  of max_i k_i.
>  The complete definition of extent is given on page 107.
> 
> but should read
> 
>  If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon 
>  is the least non-negative increment needed to round extent(Typemap) to the next multiple 
>  of max_i k_i.
> * In Fortran, whether the alignments k_i are computed according to the alignments 
> * used by the compiler in common blocks, SEQUENCE derived types, or BIND(C) derived types,
> * is implementation-dependent.
> : The complete definition of extent is given in Section 4.1.6 on page 106.
> 
> ________
> 
> mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
> p87:7-11 (=MPI-2.2 p79:7-11) read
> 
>  Rationale. The definition of extent is motivated by the assumption that the amount
>  of padding added at the end of each structure in an array of structures is the least
>  needed to fulfill alignment constraints. More explicit control of the extent is provided
>  in Section 4.1.6. Such explicit control is needed in cases where the assumption does
>  not hold, for example, where union types are used. 
>  (End of rationale.)
> 
> but should read
> 
>  Rationale. The definition of extent is motivated by the assumption that the amount
>  of padding added at the end of each structure in an array of structures is the least
>  needed to fulfill alignment constraints. More explicit control of the extent is provided
>  in Section 4.1.6. Such explicit control is needed in cases where the assumption does
>  not hold, for example, where union types are used. 
> * In Fortran, structures can be expressed with several language features, e.g.,
> * common blocks, SEQUENCE derived types, or BIND(C) derived types. The compiler may
> * use different alignments, and therefore, it is recommended to use MPI_TYPE_CREATE_RESIZED
> * for arrays of structures if an alignment may cause an alignment-gap at the end of 
> * a structure as described in Section 4.1.6 on page 106 and in Section 16.2.15 on page 580. 
>  (End of rationale.)
> 
> 
> ________
> 
> mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
> p107:13-17 (=MPI-2.2 p96:42-45) read
> 
>  If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon
>  is the least non-negative increment needed to round extent(Typemap) to the next multiple
>  of max_i k_i.
>  The formal definitions given for the various datatype constructors apply now, 
>  with the amended definition of extent.
> 
> but should read
> 
>  If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon
>  is the least non-negative increment needed to round extent(Typemap) to the next multiple
>  of max_i k_i.
> * In Fortran, whether the alignments k_i are computed according to the alignments 
> * used by the compiler in common blocks, SEQUENCE derived types, or BIND(C) derived types,
> * is implementation-dependent.
>  The formal definitions given for the various datatype constructors apply now, 
>  with the amended definition of extent.
> 
> * Rationale.
> * Before Fortran 2003, MPI_TYPE_CREATE_STRUCT could be applied to Fortran common blocks and
> * SEQUENCE derived types. With Fortran 2003, this list was extended by BIND(C) derived types
> * and MPI implementors have implemented the alignments k_i different, i.e., some based on
> * the alignments used in SEQUENCE derived types, and others according to BIND(C) derived types.
> * End of rationale.
> 
> * Advice to implementors.
> * In Fortran, it is generally recommended to use BIND(C) derived types instead of
> * common blocks or SEQUENCE derived types. Therefore it is recommended to calculate the
> * alignments k_i based on BIND(C) derived types.
> * End of advoice to implementors.
> 
> * Advice to users.
> * Structures combining different basic datatypes should be defined so that there will
> * be no gaps based on alignment rules, and if used as an array of structures, then
> * also without such an alignment-gap at the end of the structure.
> * In MPI communication, the content of such gaps would not be communicated.
> * If alignment-gaps may occur, e.g., between an odd number of floats ore REAL before
> * a double or DOUBLE PRECISION data, then such gaps should be added explicitely in 
> * the structure and also to the MPI derived datatype handle, because the communication
> * of a contiguous derived datatype may be significantly faster than the communication
> * of one that is non-contiguous due such alignment-gaps. 
> *
> * Example:
> *   Instead of
> *
> *     TYPE, BIND(C) :: my_data
> *       REAL, DIMENSION(3) :: x
> *       ! there may be a gap of the size of one REAL
> *       ! if the alignment of a DOUBLE PRECISION is 
> *       ! two times the size of a REAL
> *       DOUBLE PRECISION: p
> *     END TYPE
> * 
> * one should define
> *
> *     TYPE, BIND(C) :: my_data
> *       REAL, DIMENSION(3) :: x
> *       REAL :: gap1
> *       DOUBLE PRECISION: p
> *     END TYPE
> *
> * and also including gap1 in the matching MPI derived datatype.
> __________
> 
> mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
> p580:12,21,26,30,34,36:
> 
>  Add a forth structure entry: logical l after the double precision d
> ________
> 
> mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
> p581:11-29 read
> 
> x TODO: The following text about the extent of derived types should be
> x strongly checked by specialist on derived types!!! The correct variant should
> x be chosen.
>  To use a derived type in an array requires a correct extent of the datatype handle to
>  take care of the alignment rules applied by the compiler. These alignment rules may imply
>  that there are gaps between the elements of a derived type, and also between the array
>  elements. 
> x The alignment rules in Section 4.1 on page 85 and Section 4.1.6 on page 106
> x apply only to
> x VARIANT 1: SEQUENCE
> x VARIANT 2: BIND(C)
> x derived types. 
>  The extent of an iteroperable derived type (i.e., defined with BIND(C)) and
>  a SEQUENCE derived type with the same content may be different because C and Fortran
>  may apply different alignment rules.
> x VARIANT 1:
> x Using the SEQUENCE attribute instead of BIND(C) in the declararion on mytype, one can
> x directly use newtype to send the fooarr array.
> x VARIANT 2:
> x In the example, one can directly use newtype to send the fooarr array. The resized
> x newarrtype datatype is only needed, if mytype is a SEQUENCE derived type.
> 
> but should read
> 
>  To use a derived type in an array requires a correct extent of the datatype handle to
>  take care of the alignment rules applied by the compiler. These alignment rules may imply
>  that there are gaps between the elements of a derived type, and also between the array
>  elements.
>  The extent of an iteroperable derived type (i.e., defined with BIND(C)) and
>  a SEQUENCE derived type with the same content may be different because C and Fortran
>  may apply different alignment rules.
>  As mentioned in Section 4.1 on page 86 and Section 4.1.6 on page 106,
>  one should resize the datatype before using it for an array of structures fooarr,
>  if elements may have different alignments that may cause an alignment-gap
>  at the end of the structure.  
> _________________________
> 
> Best regards
> Rolf
> 
> 
> -- 
> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
> Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)
> _______________________________________________
> mpi3-fortran mailing list
> mpi3-fortran at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran

-- 
Jeff Squyres
jsquyres at cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/