[Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements

[Rolf Rabenseifner]

Dear all,

To understand how big and large counts should be implemented in MPI-4, 
it is important to understand the count, displacement and size model of in MPI-3.1.

As long as we do not have a common understanding of MPI-3.1, 
we will have problems to define MPI-4.

Therefore here my understanding of MPI-3. We have

- number of array elements with elements of a given type (typically represented by an MPI datatype handle).
Usual argument name: count
Usual description: number of elements in ... buffer (non-negative integer)
C-type in MPI-3.1: int
C-type in _l: MPI_Count

- an index into such an array, i.e., the number of an element.
Argument name / descriptions [Routine]:
  -- array_of_displacements / displacement ..., in multiples of oldtype extent (... integer) [MPI_TYPE_INDEXED]
  -- sdispls / integer array (of length group size). Entry j specifies the displacement ... [MPI_ALLTOALLV]
C-type in MPI-3.1: int
C-type in _l: MPI_Count

- number of bytes
Argument name / descriptions [Routine]:
  -- size / size of window in bytes (non-negative integer) [MPI_WIN_CREATE]
  -- outsize / output buffer size, in bytes (integer) [MPI_PACK_EXTERNAL]
C-type in MPI-3.1: MPI_Aint
C-type in _l:      MPI_Aint
(Wrong) C-types in MPI-3.1: int [MPI_PACK, MPI_TYPE_SIZE]
C-type (corrected) in _l:   MPI_Aint [MPI_PACK, MPI_TYPE_SIZE]

- smaller number in bytes
Some argument names: e.g., disp_unit 
Description: local unit size for displacements, in bytes (positive integer)
C-type in MPI-3.1: int
C-type in _l: still int? or MPI_Count?

- Position or relative byte displacement within an array of bytes.
Such values can be calculated as any sum and product of in, long, long long,
and MPI_Aint as long as MPI_Aint value contains a pure integer size value,
i.e., an (integer) difference of two absolute addresses within one sequential
storage, see MPI-3.1 page 115 line 31, or a MPI datatype extent, retrieved,
e.g., with MPI_TYPE_GET_EXTENT.
Argument names / description:
  -- position / current position in buffer, in bytes (integer) [MPI_PACK_EXTERNAL]
  -- array_of_displacements / byte displacement of each block (array of integer) [MPI_TYPE_CREATE_STRUCT]
C-type in MPI-3.1: MPI_Aint
C-type in _l:      MPI_Aint
(Wrong) C-types in MPI-3.1: int [MPI_PACK, MPI_ALLTOALLW]
C-type (corrected) in _l:   MPI_Aint [MPI_PACK, MPI_ALLTOALLW]

- Absolute address values for byte displacements.
These values are also valid for all byte displacements in datatype routines
and in MPI_NEIGHBOR_ALLTOALLW, provided that they are used in combination
with buffer=MPI_BOTTOM.
They cannot be used in MPI_ALLTOALLW.
With "C-type (corrected) in _l: MPI_Aint [MPI_ALLTOALLW]",
they are also usable with MPI_ALLTOALLW.


I already looked at the Large/Big Count pdf and saw that in the datatype chapter
these rules were broken, for example for the ...PACK/UNPACK... routines.


Therefore my clear question, do we agree on the rules above?


Already detected bugs in Version from Sep. 13, 2019:

- page 127:
  No idea why you changed the name from MPI_GET_ELEMENTS to MPI_TYPE_GET_ELEMENTS. 
  Should be reverted.

- MPI_PACK:
  outsize and position should be handled identical to those in MPI_PACK_EXTERNAL, i.e.,
  both are MPI_Aint...

- MPI_PACK_SIZE:
  size should be handled identical to that in MPI_PACK_EXTERNAL_SIZE, i.e.,
  MPI_Aint...

- MPI_UNPACK:
  insize and position should be handled identical to those in MPI_UNPACK_EXTERNAL, i.e.,
  both are MPI_Aint...

- MPI_Type_contiguous: the large count _l version is missing

- MPI_Type_create_darray
   -- array_of_distribs must be INTEGER, because it holds enumeration values
      and nothing else.
   -- array_of_dargs requires significant explanation because it can 
      hold an enumeration (INTEGER) and also large count values which
      can cause not understandable compiler error reports
      in Fortran, because using MPI_COUNT_KIND array_of_gsizes values
      together with an INTEGER enumeration constant, 
      here MPI_DISTRIBUTE_DFLT_DARG, would cause a compiler message
      like "no matching interface found".

      Two possible text/interface solutions:
      - If using the mpi_f08 module and MPI_DISTRIBUTE_DFLT_DARG together
        with large Count Version of this procedure, i.e., 
        INTEGER(KIND=MPI_COUNT_KIND) array_of_gsizes and array_of_dargs 
        arguments, then one should use 
           INT(MPI_DISTRIBUTE_DFLT_DARG, MPI_COUNT_KIND)
        instead of 
           MPI_DISTRIBUTE_DFLT_DARG.
      - overloading with two versions (long,normal) and (long,long)
        but i would recommend the first solution because it does
        not require additional MPI library implementation overhead.


Before I can continue with reviewing, the principles above must be
cleared/discussed/agreed/...

Best regards
Rolf



-- 
Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de .
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 .
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 .
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner .
Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .