[Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements

Tue Nov 5 11:58:31 CST 2019

Sorry, I now detected that you wanted to point me also to

>> ----- Jeff Hammond <jeff.science at gmail.com<mailto:jeff.science at gmail.com>>
>> wrote:
>> Rolf:
>> 
>> Have you looked at
>> https://github.com/mpiwg-large-count/large-count-issues/issues/6 ?

where I found your text. Thank you for remembering me.

About your choices:

> 1.change the name of MPI_Type_get_extext_x to MPI_Type_get_extext_l to fit in with all the other _l functions for large count. This is a bad idea because the MPI_Count variant is not (just) embiggening the MPI_Aint variant, it is providing a different role (target files, not absolute memory).

Agreed, bad idea.

> 2.add a new MPI_Type_get_extext_l that uses MPI_Offset. This is superfluous because we already have the _x variant with MPI_Count.

Agreed, bad idea, because there is no "int Count" Argument.

> 3.leave it well alone. This seems like the best approach.

Partially agreed, because for the  _x  Versions in the derived datatype chapter,
I would always use MPI_Count and not MPI_Offset, because one never know
for which other purpose derived datatypes may be used in the future..
The concept is not restricted to Memory and I/O.
I/O is only an example why we need to provide these _x Versions.
Therefore I prefere:

4. leave it. It is the starting point for the other _X versions.

Small diff between these two _x routines and the new ones is
that for each new one, it does not have a visible Fortran _X, 
only the overloaded one,
and each existing one must get the additional overloaded interface.
As far as I understand Fortran, there will be then only one backend routine 
with two different interfaces, because the backend is the same _X
routine for the explicit and for the overloaded MPI_Count routine.

Best regards
Rolf

----- Original Message -----
> From: "mpiwg-large-counts" <mpiwg-large-counts at lists.mpi-forum.org>
> To: "HOLMES Daniel" <d.holmes at epcc.ed.ac.uk>
> Cc: "Rolf Rabenseifner" <rabenseifner at hlrs.de>, "mpiwg-large-counts" <mpiwg-large-counts at lists.mpi-forum.org>
> Sent: Tuesday, November 5, 2019 6:10:25 PM
> Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements

> Hi Dan,
> 
> our mailers do not want to work together.
> I cannot find your "lengthy comment on that issue" in this email.
> 
> As you can see below, it is flattened.
> 
> Originally , it was with huge indents, but without any ">".
> Text looked then líke
> 
>                                        d
>                                        d
>                                        r
>                                        e
>                                        s
>                                        s
>                                        e
>                                        s
> 
> 
> In the telcon, we completely aggreed that the large count versions MPI...._l
> should keep all MPI_Aint and should add MPI_Aint where it was missing
> by being wrong (MPI_Alltoall_w and MPI_(Un)pack.
> 
> We completely agreed that all the MPI_Aint discussion is done.
> MPI_Aint is an signed integer mis-used to also store absoulte
> adresses whatever there bits may mean. And we do not change this
> nor we introduce any new such strange type.
> 
> And we agreed that there should be for the few derived datatype routines
> with MPI_Aint an additional set with MPI_Count (or MPI_Offset, which would
> be wierd) instead of MPI_Aint.
> These MPI_Count versions do not allow the use of absolute addresses.
> These MPI_Count versions are needed for the case that sizeof(MPI_Aint)
> is less than sizeof(MPI_Count), may de 4:8 or 8.12 or 8:16,
> depending on the memory size per MPI process and the filesystem size.
> 
> I only proposed in my latest email, that exactly two of them already exist:
> 
>   MPI_Type_get_extent_x and MPI_Type_get_true_extent_x
> 
> and we should keep them and take the postfix _x for this set of routines
> instead of throwing them away and reinvernting them again.
> 
> 
> For ... I would say,
> - MPI_TYPE_SIZE_X
>      the _l Version should have MPI_Aint size because it is a byte size.
> - MPI_GET_ELEMENTS_X and MPI_STATUS_SET_ELEMENTS_X
>      in my opinion, we should keep _X, and the _l version
>      is then a duplicate of it.
> 
> Best regards
> Rolf
> 
> ----- Original Message -----
>> From: "HOLMES Daniel" <d.holmes at epcc.ed.ac.uk>
>> To: "Rolf Rabenseifner" <rabenseifner at hlrs.de>
>> Cc: "mpiwg-large-counts" <mpiwg-large-counts at lists.mpi-forum.org>
>> Sent: Tuesday, November 5, 2019 3:18:13 PM
>> Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts,
>> sizes, and byte and nonbyte displacements
> 
>> Hi Rolf (et al),
>> 
>> I wrote a lengthy comment on that issue to capture my current understanding of
>> your “really wrong” assertion.
>> 
>> Broadly, we agree - I just wanted to write down the reasoning and nuances of
>> that outcome.
>> 
>> Cheers,
>> Dan.
>> —
>> Dr Daniel Holmes PhD
>> Architect (HPC Research)
>> d.holmes at epcc.ed.ac.uk<mailto:d.holmes at epcc.ed.ac.uk>
>> Phone: +44 (0) 131 651 3465
>> Mobile: +44 (0) 7940 524 088
>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT
>> —
>> The University of Edinburgh is a charitable body, registered in Scotland, with
>> registration number SC005336.
>> —
>> 
>> On 2 Nov 2019, at 09:13, Rolf Rabenseifner via mpiwg-large-counts
>> <mpiwg-large-counts at lists.mpi-forum.org<mailto:mpiwg-large-counts at lists.mpi-forum.org>>
>> wrote:
>> 
>> After the Telcon it seems that this ticket is really wrong.
>> Some or all of the routines may/should be kept.
>> And it these routines arean essential part of the future large count concept.
>> 
>> Thank you very much for pointing us to this ticket.
>> Rolf
>> 
>> ----- Jeff Hammond <jeff.science at gmail.com<mailto:jeff.science at gmail.com>>
>> wrote:
>> Rolf:
>> 
>> Have you looked at
>> https://github.com/mpiwg-large-count/large-count-issues/issues/6?
>> 
>> Jeff
>> 
>> On Fri, Nov 1, 2019 at 1:00 AM Rolf Rabenseifner <rabenseifner at hlrs.de>
>> wrote:
>> 
>> A small comment on the result of our telcon:
>> - Postfix _l for int -> MPI_Count
>> - Postfix _x for additionally
>>  MPI_Aint -> MPI_Count
>> I.e., the additional routines in the derived datatype chapter.
>> 
>> Two of them already exist
>> MPI_Type_get_(true)extent_x
>> 
>> In Fortran we will have then for the
>> same routine two aliases:
>> 
>>  - the overload one without _x
>>  - and the explicit one with _x
>> 
>> For both ones, the internal function name is the same, with _x.
>> 
>> Best regards
>> Rolf
>> 
>> 
>> ----- Jeff Hammond <jeff.science at gmail.com> wrote:
>> On Thu, Oct 31, 2019 at 7:48 AM Rolf Rabenseifner <rabenseifner at hlrs.de>
>> wrote:
>> 
>> Dear all,
>> 
>> here my summary as input for our telcon today.
>> 
>> In principle, it is a very simple question:
>> 
>> with large Counts, do we
>> - keep all MPI_Aint
>> - or do we substitute MPI_Aint by MPI_Count?
>> 
>> 
>> I haven't been involved as much lately but did we not use MPI_Count for
>> count and element displacements in the large count proposal?  We need to
>> use MPI_Aint for offsets into memory because that is what this type is
>> for.
>> 
>> Jeff
>> 
>> 
>> 
>> In principle, the MPI Forum answered this question already
>> for MPI-3.0 in 2012 with a clear YES:
>> 
>> int MPI_Type_get_extent(MPI_Datatype datatype,
>>     MPI_Aint *lb,  MPI_Aint *extent)
>> int MPI_Type_get_extent_x(MPI_Datatype datatype,
>>     MPI_Count *lb, MPI_Count *extent)
>> 
>> About Jeff H. question:
>> If we limit the API to not support MPI_Count
>> means that an MPI implementation has not really such quality options
>> when using I/O fileviews, because the API is restricted to
>> MPI_Aint (which should be implemented based on the, e.g.,
>> 64bit memory system).
>> 
>> About Jim's comment:
>> 
>> Apologies, it's been a while since I looked at the I/O interfaces.
>> If
>> I/O
>> only needs relative displacements that have normal integer
>> semantics,
>> then
>> I don't see why MPI_Count would not work for this purpose. If you
>> have
>> an
>> MPI_Aint that contains a relative displacement, it also has normal
>> integer
>> semantics and can be converted to an MPI_Count.
>> 
>> Yes, but this automatically implies that the datatypes must also
>> be able to handle MPI_Count.
>> 
>> The only case we really
>> need to look out for is when an integer type contains an absolute
>> address.
>> In those cases, the quantity in the variable cannot be treated as a
>> normal
>> integer and we need special routines to work with it.
>> 
>> Yes, this happens when we extend MPI_Aint in the derived datatype
>> routines
>> to MPI_Count.
>> 
>> But in principle, this is not a big Problem, as you all could see in
>> the previous emails:
>> 
>> - We must do for MPI_Count the same as we did for MPI_Aint,
>> i.e., we'll have long versions of the routines
>>  MPI_Get_address, MPI_Aint_diff, MPI_Aint_add
>> 
>> - And we must ensure that the type cast from MPI_Aint to
>> MPI_Count works, which is a small new advice to implementors
>> for MPI_Det_address.
>> 
>> Therefore again my 4 questions:
>> 
>> - Should the new large count routines be prepared for
>> more than 10 or 20 Exabyte files where we need 64/65 or
>> or 65/66 unsigned/signed integers for relative byte
>> displacements or byte counts?
>> If yes, then all MPI_Aint arguments must be substituted by MPI_Count.
>> 
>> (In other words, do we want to be prepared for another 25 years of
>> MPI?
>> :-)
>> 
>> As stated above, the MPI-Forum already decided 2012 with a YES.
>> 
>> - Should we allow that these new routines are also used for memory
>> description,
>> where we typically need only the large MPI_Count "count" arguments?
>> (or should we provide two different new routines for each routine
>> that
>>  currently has int Count/... and MPI_Aint disp/... arguments)
>> 
>> I expect, that nobody wants to have two different large versions of
>> for example MPI_Type_create_struct.
>> 
>> - Should we allow a mix of old and new routines, especially for
>> memory-based
>> usage, that old-style MPI_Get_address is used to retrieve an absolute
>> address and then, e.g., new style MPI_Type_create_struct with
>> MPI_Count blocklength and displacements is used?
>> 
>> I expect that forbidding such a mix would be a problem for Software
>> development.
>> Often old-style modules must work together with new-style modules.
>> 
>> - Do we want to require for this type cast of MPI_Aint addr into
>> MPI_Count
>> that it is allowed to do this cast with a normal assignment, rather
>> than
>> a special MPI function?
>> 
>> I expect yes, because for must usage of MPI_Aint and MPI_Count,
>> it is for relative displacements or byte counts, i.e. for normal
>> integers and therefore automatic type cast between MPI_Aint
>> and MPI_Count is a must.
>> 
>> With yes to all four questions, the proposed solution above is
>> the easiest way.
>> 
>> Hope to see/hear you today in our telcon.
>> 
>> Best regards
>> Rolf

-- 
Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de .
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 .
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 .
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner .
Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .