[MPI3 Fortran] Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION

Thu Oct 6 18:55:13 CDT 2011

Dear all,

here is my proposal to solve the alignment problems.
Please check carefully.

It is touching only the Fortran side.
C problems with alignments are not really touched, but should be small.

----------------------------------------------------------------------

(Lines with * are new, lines with : are modified, lines with x are removed)

mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
p86:45-47 (=MPI-2.2 p78:45-47) read

  If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon 
  is the least non-negative increment needed to round extent(Typemap) to the next multiple 
  of max_i k_i.
  The complete definition of extent is given on page 107.

but should read

  If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon 
  is the least non-negative increment needed to round extent(Typemap) to the next multiple 
  of max_i k_i.
* In Fortran, whether the alignments k_i are computed according to the alignments 
* used by the compiler in common blocks, SEQUENCE derived types, or BIND(C) derived types,
* is implementation-dependent.
: The complete definition of extent is given in Section 4.1.6 on page 106.

________

mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
p87:7-11 (=MPI-2.2 p79:7-11) read

  Rationale. The definition of extent is motivated by the assumption that the amount
  of padding added at the end of each structure in an array of structures is the least
  needed to fulfill alignment constraints. More explicit control of the extent is provided
  in Section 4.1.6. Such explicit control is needed in cases where the assumption does
  not hold, for example, where union types are used. 
  (End of rationale.)

but should read

  Rationale. The definition of extent is motivated by the assumption that the amount
  of padding added at the end of each structure in an array of structures is the least
  needed to fulfill alignment constraints. More explicit control of the extent is provided
  in Section 4.1.6. Such explicit control is needed in cases where the assumption does
  not hold, for example, where union types are used. 
* In Fortran, structures can be expressed with several language features, e.g.,
* common blocks, SEQUENCE derived types, or BIND(C) derived types. The compiler may
* use different alignments, and therefore, it is recommended to use MPI_TYPE_CREATE_RESIZED
* for arrays of structures if an alignment may cause an alignment-gap at the end of 
* a structure as described in Section 4.1.6 on page 106 and in Section 16.2.15 on page 580. 
  (End of rationale.)

________

mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
p107:13-17 (=MPI-2.2 p96:42-45) read

  If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon
  is the least non-negative increment needed to round extent(Typemap) to the next multiple
  of max_i k_i.
  The formal definitions given for the various datatype constructors apply now, 
  with the amended definition of extent.

but should read

  If type_i requires alignment to a byte address that is a multiple of k_i, then epsilon
  is the least non-negative increment needed to round extent(Typemap) to the next multiple
  of max_i k_i.
* In Fortran, whether the alignments k_i are computed according to the alignments 
* used by the compiler in common blocks, SEQUENCE derived types, or BIND(C) derived types,
* is implementation-dependent.
  The formal definitions given for the various datatype constructors apply now, 
  with the amended definition of extent.

* Rationale.
* Before Fortran 2003, MPI_TYPE_CREATE_STRUCT could be applied to Fortran common blocks and
* SEQUENCE derived types. With Fortran 2003, this list was extended by BIND(C) derived types
* and MPI implementors have implemented the alignments k_i different, i.e., some based on
* the alignments used in SEQUENCE derived types, and others according to BIND(C) derived types.
* End of rationale.

* Advice to implementors.
* In Fortran, it is generally recommended to use BIND(C) derived types instead of
* common blocks or SEQUENCE derived types. Therefore it is recommended to calculate the
* alignments k_i based on BIND(C) derived types.
* End of advoice to implementors.

* Advice to users.
* Structures combining different basic datatypes should be defined so that there will
* be no gaps based on alignment rules, and if used as an array of structures, then
* also without such an alignment-gap at the end of the structure.
* In MPI communication, the content of such gaps would not be communicated.
* If alignment-gaps may occur, e.g., between an odd number of floats ore REAL before
* a double or DOUBLE PRECISION data, then such gaps should be added explicitely in 
* the structure and also to the MPI derived datatype handle, because the communication
* of a contiguous derived datatype may be significantly faster than the communication
* of one that is non-contiguous due such alignment-gaps. 
*
* Example:
*   Instead of
*
*     TYPE, BIND(C) :: my_data
*       REAL, DIMENSION(3) :: x
*       ! there may be a gap of the size of one REAL
*       ! if the alignment of a DOUBLE PRECISION is 
*       ! two times the size of a REAL
*       DOUBLE PRECISION: p
*     END TYPE
* 
* one should define
*
*     TYPE, BIND(C) :: my_data
*       REAL, DIMENSION(3) :: x
*       REAL :: gap1
*       DOUBLE PRECISION: p
*     END TYPE
*
* and also including gap1 in the matching MPI derived datatype.
__________

mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
p580:12,21,26,30,34,36:

  Add a forth structure entry: logical l after the double precision d
________

mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
p581:11-29 read

x TODO: The following text about the extent of derived types should be
x strongly checked by specialist on derived types!!! The correct variant should
x be chosen.
  To use a derived type in an array requires a correct extent of the datatype handle to
  take care of the alignment rules applied by the compiler. These alignment rules may imply
  that there are gaps between the elements of a derived type, and also between the array
  elements. 
x The alignment rules in Section 4.1 on page 85 and Section 4.1.6 on page 106
x apply only to
x VARIANT 1: SEQUENCE
x VARIANT 2: BIND(C)
x derived types. 
  The extent of an iteroperable derived type (i.e., defined with BIND(C)) and
  a SEQUENCE derived type with the same content may be different because C and Fortran
  may apply different alignment rules.
x VARIANT 1:
x Using the SEQUENCE attribute instead of BIND(C) in the declararion on mytype, one can
x directly use newtype to send the fooarr array.
x VARIANT 2:
x In the example, one can directly use newtype to send the fooarr array. The resized
x newarrtype datatype is only needed, if mytype is a SEQUENCE derived type.

but should read

  To use a derived type in an array requires a correct extent of the datatype handle to
  take care of the alignment rules applied by the compiler. These alignment rules may imply
  that there are gaps between the elements of a derived type, and also between the array
  elements.
  The extent of an iteroperable derived type (i.e., defined with BIND(C)) and
  a SEQUENCE derived type with the same content may be different because C and Fortran
  may apply different alignment rules.
  As mentioned in Section 4.1 on page 86 and Section 4.1.6 on page 106,
  one should resize the datatype before using it for an array of structures fooarr,
  if elements may have different alignments that may cause an alignment-gap
  at the end of the structure.  
_________________________

Best regards
Rolf

-- 
Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)