[MPI3 Fortran] Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION

Rolf Rabenseifner rabenseifner at hlrs.de
Sun Oct 9 01:11:48 CDT 2011


Jeff, 
your mail didn't contain an attachment, nor the text wit "> " was modified,
nor there have been lines without "> " showing the grammer corrections.
I should have them as soon as possible.
Best regards
Rolf

----- Original Message -----
> From: "Jeff Squyres" <jsquyres at cisco.com>
> To: "MPI-3 Fortran working group" <mpi3-fortran at lists.mpi-forum.org>
> Sent: Friday, October 7, 2011 4:48:17 PM
> Subject: Re: [MPI3 Fortran] Serious problem/bug in MPI libraries with the alignment of MPI_DOUBLE_PRECISION
> I guess this looks ok -- there's some grammar changes I'd make,
> though.
> 
> One thing to be careful of:
> 
> > * Advice to users.
> > * Structures combining different basic datatypes should be defined
> > so that there will
> > * be no gaps based on alignment rules, and if used as an array of
> > structures, then
> > * also without such an alignment-gap at the end of the structure.
> > * In MPI communication, the content of such gaps would not be
> > communicated.
> 
> 
> ^^^ This is not necessarily true. A smart enough MPI implementation
> may realize that it's faster to treat a small gap as a contiguous
> datatype and just ignore the extra data, especially if it's in a
> homogeneous environment, and copy-in/copy-out is used anyway.
> 
> 
> 
> On Oct 6, 2011, at 7:55 PM, Rolf Rabenseifner wrote:
> 
> > Dear all,
> >
> > here is my proposal to solve the alignment problems.
> > Please check carefully.
> >
> > It is touching only the Fortran side.
> > C problems with alignments are not really touched, but should be
> > small.
> >
> > ----------------------------------------------------------------------
> >
> > (Lines with * are new, lines with : are modified, lines with x are
> > removed)
> >
> > mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
> > p86:45-47 (=MPI-2.2 p78:45-47) read
> >
> >  If type_i requires alignment to a byte address that is a multiple
> >  of k_i, then epsilon
> >  is the least non-negative increment needed to round extent(Typemap)
> >  to the next multiple
> >  of max_i k_i.
> >  The complete definition of extent is given on page 107.
> >
> > but should read
> >
> >  If type_i requires alignment to a byte address that is a multiple
> >  of k_i, then epsilon
> >  is the least non-negative increment needed to round extent(Typemap)
> >  to the next multiple
> >  of max_i k_i.
> > * In Fortran, whether the alignments k_i are computed according to
> > the alignments
> > * used by the compiler in common blocks, SEQUENCE derived types, or
> > BIND(C) derived types,
> > * is implementation-dependent.
> > : The complete definition of extent is given in Section 4.1.6 on
> > page 106.
> >
> > ________
> >
> > mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
> > p87:7-11 (=MPI-2.2 p79:7-11) read
> >
> >  Rationale. The definition of extent is motivated by the assumption
> >  that the amount
> >  of padding added at the end of each structure in an array of
> >  structures is the least
> >  needed to fulfill alignment constraints. More explicit control of
> >  the extent is provided
> >  in Section 4.1.6. Such explicit control is needed in cases where
> >  the assumption does
> >  not hold, for example, where union types are used.
> >  (End of rationale.)
> >
> > but should read
> >
> >  Rationale. The definition of extent is motivated by the assumption
> >  that the amount
> >  of padding added at the end of each structure in an array of
> >  structures is the least
> >  needed to fulfill alignment constraints. More explicit control of
> >  the extent is provided
> >  in Section 4.1.6. Such explicit control is needed in cases where
> >  the assumption does
> >  not hold, for example, where union types are used.
> > * In Fortran, structures can be expressed with several language
> > features, e.g.,
> > * common blocks, SEQUENCE derived types, or BIND(C) derived types.
> > The compiler may
> > * use different alignments, and therefore, it is recommended to use
> > MPI_TYPE_CREATE_RESIZED
> > * for arrays of structures if an alignment may cause an
> > alignment-gap at the end of
> > * a structure as described in Section 4.1.6 on page 106 and in
> > Section 16.2.15 on page 580.
> >  (End of rationale.)
> >
> >
> > ________
> >
> > mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
> > p107:13-17 (=MPI-2.2 p96:42-45) read
> >
> >  If type_i requires alignment to a byte address that is a multiple
> >  of k_i, then epsilon
> >  is the least non-negative increment needed to round extent(Typemap)
> >  to the next multiple
> >  of max_i k_i.
> >  The formal definitions given for the various datatype constructors
> >  apply now,
> >  with the amended definition of extent.
> >
> > but should read
> >
> >  If type_i requires alignment to a byte address that is a multiple
> >  of k_i, then epsilon
> >  is the least non-negative increment needed to round extent(Typemap)
> >  to the next multiple
> >  of max_i k_i.
> > * In Fortran, whether the alignments k_i are computed according to
> > the alignments
> > * used by the compiler in common blocks, SEQUENCE derived types, or
> > BIND(C) derived types,
> > * is implementation-dependent.
> >  The formal definitions given for the various datatype constructors
> >  apply now,
> >  with the amended definition of extent.
> >
> > * Rationale.
> > * Before Fortran 2003, MPI_TYPE_CREATE_STRUCT could be applied to
> > Fortran common blocks and
> > * SEQUENCE derived types. With Fortran 2003, this list was extended
> > by BIND(C) derived types
> > * and MPI implementors have implemented the alignments k_i
> > different, i.e., some based on
> > * the alignments used in SEQUENCE derived types, and others
> > according to BIND(C) derived types.
> > * End of rationale.
> >
> > * Advice to implementors.
> > * In Fortran, it is generally recommended to use BIND(C) derived
> > types instead of
> > * common blocks or SEQUENCE derived types. Therefore it is
> > recommended to calculate the
> > * alignments k_i based on BIND(C) derived types.
> > * End of advoice to implementors.
> >
> > * Advice to users.
> > * Structures combining different basic datatypes should be defined
> > so that there will
> > * be no gaps based on alignment rules, and if used as an array of
> > structures, then
> > * also without such an alignment-gap at the end of the structure.
> > * In MPI communication, the content of such gaps would not be
> > communicated.
> > * If alignment-gaps may occur, e.g., between an odd number of floats
> > ore REAL before
> > * a double or DOUBLE PRECISION data, then such gaps should be added
> > explicitely in
> > * the structure and also to the MPI derived datatype handle, because
> > the communication
> > * of a contiguous derived datatype may be significantly faster than
> > the communication
> > * of one that is non-contiguous due such alignment-gaps.
> > *
> > * Example:
> > * Instead of
> > *
> > * TYPE, BIND(C) :: my_data
> > * REAL, DIMENSION(3) :: x
> > * ! there may be a gap of the size of one REAL
> > * ! if the alignment of a DOUBLE PRECISION is
> > * ! two times the size of a REAL
> > * DOUBLE PRECISION: p
> > * END TYPE
> > *
> > * one should define
> > *
> > * TYPE, BIND(C) :: my_data
> > * REAL, DIMENSION(3) :: x
> > * REAL :: gap1
> > * DOUBLE PRECISION: p
> > * END TYPE
> > *
> > * and also including gap1 in the matching MPI derived datatype.
> > __________
> >
> > mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
> > p580:12,21,26,30,34,36:
> >
> >  Add a forth structure entry: logical l after the double precision d
> > ________
> >
> > mpi-report-F2008-2011-09-08-changeonlyplustickets.pdf
> > p581:11-29 read
> >
> > x TODO: The following text about the extent of derived types should
> > be
> > x strongly checked by specialist on derived types!!! The correct
> > variant should
> > x be chosen.
> >  To use a derived type in an array requires a correct extent of the
> >  datatype handle to
> >  take care of the alignment rules applied by the compiler. These
> >  alignment rules may imply
> >  that there are gaps between the elements of a derived type, and
> >  also between the array
> >  elements.
> > x The alignment rules in Section 4.1 on page 85 and Section 4.1.6 on
> > page 106
> > x apply only to
> > x VARIANT 1: SEQUENCE
> > x VARIANT 2: BIND(C)
> > x derived types.
> >  The extent of an iteroperable derived type (i.e., defined with
> >  BIND(C)) and
> >  a SEQUENCE derived type with the same content may be different
> >  because C and Fortran
> >  may apply different alignment rules.
> > x VARIANT 1:
> > x Using the SEQUENCE attribute instead of BIND(C) in the declararion
> > on mytype, one can
> > x directly use newtype to send the fooarr array.
> > x VARIANT 2:
> > x In the example, one can directly use newtype to send the fooarr
> > array. The resized
> > x newarrtype datatype is only needed, if mytype is a SEQUENCE
> > derived type.
> >
> > but should read
> >
> >  To use a derived type in an array requires a correct extent of the
> >  datatype handle to
> >  take care of the alignment rules applied by the compiler. These
> >  alignment rules may imply
> >  that there are gaps between the elements of a derived type, and
> >  also between the array
> >  elements.
> >  The extent of an iteroperable derived type (i.e., defined with
> >  BIND(C)) and
> >  a SEQUENCE derived type with the same content may be different
> >  because C and Fortran
> >  may apply different alignment rules.
> >  As mentioned in Section 4.1 on page 86 and Section 4.1.6 on page
> >  106,
> >  one should resize the datatype before using it for an array of
> >  structures fooarr,
> >  if elements may have different alignments that may cause an
> >  alignment-gap
> >  at the end of the structure.
> > _________________________
> >
> > Best regards
> > Rolf
> >
> >
> > --
> > Dr. Rolf Rabenseifner . . . . . . . . . .. email
> > rabenseifner at hlrs.de
> > High Performance Computing Center (HLRS) . phone
> > ++49(0)711/685-65530
> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 /
> > 685-65832
> > Head of Dpmt Parallel Computing . . .
> > www.hlrs.de/people/rabenseifner
> > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)
> > _______________________________________________
> > mpi3-fortran mailing list
> > mpi3-fortran at lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran
> 
> 
> --
> Jeff Squyres
> jsquyres at cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> 
> _______________________________________________
> mpi3-fortran mailing list
> mpi3-fortran at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran

-- 
Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)



More information about the mpiwg-fortran mailing list