[Mpiwg-large-counts] A perfect trick in C with similar API as with Fortran overloading - instead of the many new _l functions in C

Mon Oct 21 13:40:01 CDT 2019

Dear all,

Yes, I understand. 

And providing the new API in a new module mpi_x.h, but with same function names
MPI_Send ... is then also an evil.
Especially because the two MPI_Send(..,int count,..) from mpi.h 
and MPI_Send(..,MPI_Count count,..) from mpi_x.h cannot be together 
within the same lib because C does not allow to define an ABI-function-Name
that is not identical to the API-function-name.

Therefore next proposal:

We do what we planned, but can additionally provide an mpi_l.h header file
containing only a set of aliases
#define MPI_Send(...) MPI_Send_l(...)
...

User who changed somehow all their 4 byte integer in into 8 byte integer 
can then directly use the new routines without changing all the MPI function calls.

As far as I understand, wouldn't this add-on proposal fall into the evil category,
or would it?

Best regards
Rolf

----- Original Message -----
> From: "Jeff Squyres" <jsquyres at cisco.com>
> To: "mpiwg-large-counts" <mpiwg-large-counts at lists.mpi-forum.org>
> Cc: "Rolf Rabenseifner" <rabenseifner at hlrs.de>, "Jeff Hammond" <jeff.science at gmail.com>
> Sent: Monday, October 21, 2019 3:41:53 PM
> Subject: Re: [Mpiwg-large-counts] A perfect trick in C with similar API as with Fortran overloading - instead of the
> many new _l functions in C

> I agree: breaking ABI in this way is truly evil.
> 
> Specifically: compile your app with implementation version X.  Then run your app
> with same implementation, but version Y (this is not uncommon for commercial
> ISV MPI apps).  Things fail in weird, non-obvious ways (or worse, they *don't*
> fail, but silently give you wrong results).
> 
> Speaking as someone who writes code for a living, changing the signature of an
> existing, well-established function is evil.  Changing it in ways that a
> compiler may not even warn you about (in forward- and backward-compatility
> scenarios) is TRULY evil.
> 
> 
> 
> 
>> On Oct 21, 2019, at 9:35 AM, Jeff Hammond via mpiwg-large-counts
>> <mpiwg-large-counts at lists.mpi-forum.org> wrote:
>> 
>> ABI is very much a user problem. We are not breaking ABI for this.
>> 
>> What is being done now is five years in the making. I’m not sure why you didn’t
>> comment until recently.
>> 
>> Jeff
>> 
>> Sent from my iPhone
>> 
>>> On Oct 20, 2019, at 10:49 PM, Rolf Rabenseifner via mpiwg-large-counts
>>> <mpiwg-large-counts at lists.mpi-forum.org> wrote:
>>> 
>>> Dear Tony and all,
>>> 
>>>> the group has labored over each area... 1 sided has had the least attention so
>>>> far... All of MPI is to be made 64 bit clean not parts.
>>> 
>>> My proposal ia for whole MPI.
>>> Only for few routines we need some additional _x versions.
>>> 
>>>> Partial answer to your earliest question in the note: Changing the meaning of
>>>> existing APIs was disallowed some time ago.  Certainly, for in arguments of
>>>> count , replacing int with MPI_Count is a partial solution.  But it changes the
>>>> APIs ...
>>> 
>>> The goal is backward compatibility.
>>> And this one is provided.
>>> 
>>>> In C, a program written for the assumption of big count and compiled
>>>> accidentally with an MPI-3 compliant MPI will silently build and fail at
>>>> runtime ... rounding of integers ...
>>> 
>>> Let's recommend to the users of the long coutnt feature that they should at a
>>> check of the version. In C, in can be tested at compile time with cpp.
>>> 
>>>> and changing the AbI violates the ABI for tools and such  — two  possible
>>>> reasons why not chosen ... but we will have to go refresh on all the reasons .
>>> 
>>> Not a user problem ...
>>> 
>>>> I am sure you will get extensive feedback on these questions from the whole
>>>> group ? :-)
>>> 
>>> And also by the plenum.
>>> 
>>> Best regards
>>> Rolf
>>> 
>>> ----- Tony Skjellum <skjellum at gmail.com> wrote:
>>>> Rolf, the group has labored over each area... 1 sided has had the least
>>>> attention so far... All of MPI is to be made 64 bit clean not parts.
>>>> 
>>>> Partial answer to your earliest question in the note: Changing the meaning of
>>>> existing APIs was disallowed some time ago.  Certainly, for in arguments of
>>>> count , replacing int with MPI_Count is a partial solution.  But it changes the
>>>> APIs ...
>>>> 
>>>> In C, a program written for the assumption of big count and compiled
>>>> accidentally with an MPI-3 compliant MPI will silently build and fail at
>>>> runtime ... rounding of integers ... and changing the AbI violates the ABI for
>>>> tools and such  — two  possible reasons why not chosen ... but we will have to
>>>> go refresh on all the reasons .
>>>> 
>>>> I am sure you will get extensive feedback on these questions from the whole
>>>> group ? :-)
>>>> 
>>>> Tony
>>>> 
>>>> Anthony Skjellum, PhD
>>>> 205-807-4968
>>>> 
>>>> 
>>>>> On Oct 20, 2019, at 2:25 PM, Rolf Rabenseifner via mpiwg-large-counts
>>>>> <mpiwg-large-counts at lists.mpi-forum.org> wrote:
>>>>> 
>>>>> Dear all,
>>>>> 
>>>>> a participant in my latest course asked me, whether we not
>>>>> substituting in by MPI_Count in all "int xxxxx" input arguments.
>>>>> The compiler automatically would cast to MPI_Count which may be a fast
>>>>> instruction. Then we need the new _x or _l functions only for those
>>>>> rare functions with MPI_Count as output or array argument.
>>>>> 
>>>>> Was this ever discussed?
>>>>> 
>>>>> Second question:
>>>>> Does the large Count roup plan, only to use MPI_Count for data,
>>>>> i.e., only in communication, I/O, and derived type routines?
>>>>> 
>>>>> Is it correct, that we keep the number of
>>>>> - request handles in request arrays,
>>>>> - ranks an size in/of communicator and group handles,
>>>>> will stay int?
>>>>> 
>>>>> How do we handle disp in 1-sided communication?
>>>>> 
>>>>> With this approach, it seems that explicit MPI_Count versions of
>>>>> a subroutine is needed only for
>>>>> MPI_[I][NEIGHBOR_]{ALLTOALL|ALLGATHER}{V|W}
>>>>> MPI_[I]{GATHER|SCATTER}V
>>>>> MPI_[I]REDUCE_SCATTER
>>>>> (18 routines)
>>>>> MPI_WIN_SHARED_QUERY --> *disp
>>>>> (1 Routine)
>>>>> MPI_[UN]PACK
>>>>> MPI_PACK_SIZE _x/_l version should have MPI_Aint (same as for
>>>>> MPI_...PACK..._external)
>>>>> (3 routines)
>>>>> MPI_GET_COUNT
>>>>> MPI_GET_ELEMENTS (has already an _X version)
>>>>> MPI_TYPE_SIZE (has already an _X version)
>>>>> 
>>>>> Which solution exists for MPI_CONVERSION_FN_NULL?
>>>>> 
>>>>> All other 100 routines with "int count" and "int size" seems
>>>>> to be solved with directly changing it into "MPI_Count count" and "MPI_Count
>>>>> size".
>>>>> 
>>>>> Best regards
>>>>> Rolf
>>>>> 
>>>>> ----- Original Message -----
>>>>>> From: "mpiwg-large-counts" <mpiwg-large-counts at lists.mpi-forum.org>
>>>>>> To: "mpiwg-large-counts" <mpiwg-large-counts at lists.mpi-forum.org>
>>>>>> Cc: "Jeff Squyres" <jsquyres at cisco.com>
>>>>>> Sent: Saturday, October 19, 2019 10:26:23 PM
>>>>>> Subject: [Mpiwg-large-counts] Pythonizing instructions
>>>>> 
>>>>>> As discussed in our last meeting, I've written a first take at instructions for
>>>>>> Chapter Authors for how to write Pythonized functions.
>>>>>> 
>>>>>> These instructions are not yet intended for a wide audience -- they are intended
>>>>>> solely for the working group.  Let's iterate on them here in the WG and get
>>>>>> them to a good enough state before sharing them widely:
>>>>>> 
>>>>>> https://github.com/mpiwg-large-count/large-count-issues/wiki/pythonizing
>>>>>> 
>>>>>> Puri/Tony/Dan: please try to use these instructions to Pythonize a few bindings.
>>>>>> The PR for the Pythonization is
>>>>>> https://github.com/mpiwg-large-count/mpi-standard/pull/2.  All even-numbered
>>>>>> chapters are done, and a small number of odd-numbered chapters.  Grep for
>>>>>> `funcdef` in chapter *.tex files to find a chapter that hasn't been Pythonized
>>>>>> yet (funcdef is the LaTeX macro for hard-coded LaTeX macros).
>>>>>> 
>>>>>> Send your feedback here and/or we'll discuss on Wednesday.
>>>>>> 
>>>>>> --
>>>>>> Jeff Squyres
>>>>>> jsquyres at cisco.com
>>>>>> 
>>>>>> _______________________________________________
>>>>>> mpiwg-large-counts mailing list
>>>>>> mpiwg-large-counts at lists.mpi-forum.org
>>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts
>>>>> 
>>>>> --
>>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de .
>>>>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 .
>>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 .
>>>>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner .
>>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .
>>>>> _______________________________________________
>>>>> mpiwg-large-counts mailing list
>>>>> mpiwg-large-counts at lists.mpi-forum.org
>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts
>>> 
>>> --
>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de .
>>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 .
>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 .
>>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner .
>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .
>>> _______________________________________________
>>> mpiwg-large-counts mailing list
>>> mpiwg-large-counts at lists.mpi-forum.org
>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts
>> _______________________________________________
>> mpiwg-large-counts mailing list
>> mpiwg-large-counts at lists.mpi-forum.org
>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts
> 
> 
> --
> Jeff Squyres
> jsquyres at cisco.com

-- 
Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de .
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 .
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 .
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner .
Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .