[Mpiwg-large-counts] A perfect trick in C with similar API as with Fortran overloading - instead of the many new _l functions in C

Mon Oct 21 13:43:26 CDT 2019

If users want such a preprocessor solution, they should get it from a third-party implementation. I do not think it belongs in the MPI standard. 

Jeff

> On Oct 21, 2019, at 11:40 AM, Rolf Rabenseifner <rabenseifner at hlrs.de> wrote:
> 
> Dear all,
> 
> Yes, I understand. 
> 
> And providing the new API in a new module mpi_x.h, but with same function names
> MPI_Send ... is then also an evil.
> Especially because the two MPI_Send(..,int count,..) from mpi.h 
> and MPI_Send(..,MPI_Count count,..) from mpi_x.h cannot be together 
> within the same lib because C does not allow to define an ABI-function-Name
> that is not identical to the API-function-name.
> 
> Therefore next proposal:
> 
> We do what we planned, but can additionally provide an mpi_l.h header file
> containing only a set of aliases
> #define MPI_Send(...) MPI_Send_l(...)
> ...
> 
> User who changed somehow all their 4 byte integer in into 8 byte integer 
> can then directly use the new routines without changing all the MPI function calls.
> 
> As far as I understand, wouldn't this add-on proposal fall into the evil category,
> or would it?
> 
> Best regards
> Rolf
> 
> 
> ----- Original Message -----
>> From: "Jeff Squyres" <jsquyres at cisco.com>
>> To: "mpiwg-large-counts" <mpiwg-large-counts at lists.mpi-forum.org>
>> Cc: "Rolf Rabenseifner" <rabenseifner at hlrs.de>, "Jeff Hammond" <jeff.science at gmail.com>
>> Sent: Monday, October 21, 2019 3:41:53 PM
>> Subject: Re: [Mpiwg-large-counts] A perfect trick in C with similar API as with Fortran overloading - instead of the
>> many new _l functions in C
> 
>> I agree: breaking ABI in this way is truly evil.
>> 
>> Specifically: compile your app with implementation version X.  Then run your app
>> with same implementation, but version Y (this is not uncommon for commercial
>> ISV MPI apps).  Things fail in weird, non-obvious ways (or worse, they *don't*
>> fail, but silently give you wrong results).
>> 
>> Speaking as someone who writes code for a living, changing the signature of an
>> existing, well-established function is evil.  Changing it in ways that a
>> compiler may not even warn you about (in forward- and backward-compatility
>> scenarios) is TRULY evil.
>> 
>> 
>> 
>> 
>>> On Oct 21, 2019, at 9:35 AM, Jeff Hammond via mpiwg-large-counts
>>> <mpiwg-large-counts at lists.mpi-forum.org> wrote:
>>> 
>>> ABI is very much a user problem. We are not breaking ABI for this.
>>> 
>>> What is being done now is five years in the making. I’m not sure why you didn’t
>>> comment until recently.
>>> 
>>> Jeff
>>> 
>>> Sent from my iPhone
>>> 
>>>> On Oct 20, 2019, at 10:49 PM, Rolf Rabenseifner via mpiwg-large-counts
>>>> <mpiwg-large-counts at lists.mpi-forum.org> wrote:
>>>> 
>>>> Dear Tony and all,
>>>> 
>>>>> the group has labored over each area... 1 sided has had the least attention so
>>>>> far... All of MPI is to be made 64 bit clean not parts.
>>>> 
>>>> My proposal ia for whole MPI.
>>>> Only for few routines we need some additional _x versions.
>>>> 
>>>>> Partial answer to your earliest question in the note: Changing the meaning of
>>>>> existing APIs was disallowed some time ago.  Certainly, for in arguments of
>>>>> count , replacing int with MPI_Count is a partial solution.  But it changes the
>>>>> APIs ...
>>>> 
>>>> The goal is backward compatibility.
>>>> And this one is provided.
>>>> 
>>>>> In C, a program written for the assumption of big count and compiled
>>>>> accidentally with an MPI-3 compliant MPI will silently build and fail at
>>>>> runtime ... rounding of integers ...
>>>> 
>>>> Let's recommend to the users of the long coutnt feature that they should at a
>>>> check of the version. In C, in can be tested at compile time with cpp.
>>>> 
>>>>> and changing the AbI violates the ABI for tools and such  — two  possible
>>>>> reasons why not chosen ... but we will have to go refresh on all the reasons .
>>>> 
>>>> Not a user problem ...
>>>> 
>>>>> I am sure you will get extensive feedback on these questions from the whole
>>>>> group ? :-)
>>>> 
>>>> And also by the plenum.
>>>> 
>>>> Best regards
>>>> Rolf
>>>> 
>>>> ----- Tony Skjellum <skjellum at gmail.com> wrote:
>>>>> Rolf, the group has labored over each area... 1 sided has had the least
>>>>> attention so far... All of MPI is to be made 64 bit clean not parts.
>>>>> 
>>>>> Partial answer to your earliest question in the note: Changing the meaning of
>>>>> existing APIs was disallowed some time ago.  Certainly, for in arguments of
>>>>> count , replacing int with MPI_Count is a partial solution.  But it changes the
>>>>> APIs ...
>>>>> 
>>>>> In C, a program written for the assumption of big count and compiled
>>>>> accidentally with an MPI-3 compliant MPI will silently build and fail at
>>>>> runtime ... rounding of integers ... and changing the AbI violates the ABI for
>>>>> tools and such  — two  possible reasons why not chosen ... but we will have to
>>>>> go refresh on all the reasons .
>>>>> 
>>>>> I am sure you will get extensive feedback on these questions from the whole
>>>>> group ? :-)
>>>>> 
>>>>> Tony
>>>>> 
>>>>> Anthony Skjellum, PhD
>>>>> 205-807-4968
>>>>> 
>>>>> 
>>>>>> On Oct 20, 2019, at 2:25 PM, Rolf Rabenseifner via mpiwg-large-counts
>>>>>> <mpiwg-large-counts at lists.mpi-forum.org> wrote:
>>>>>> 
>>>>>> Dear all,
>>>>>> 
>>>>>> a participant in my latest course asked me, whether we not
>>>>>> substituting in by MPI_Count in all "int xxxxx" input arguments.
>>>>>> The compiler automatically would cast to MPI_Count which may be a fast
>>>>>> instruction. Then we need the new _x or _l functions only for those
>>>>>> rare functions with MPI_Count as output or array argument.
>>>>>> 
>>>>>> Was this ever discussed?
>>>>>> 
>>>>>> Second question:
>>>>>> Does the large Count roup plan, only to use MPI_Count for data,
>>>>>> i.e., only in communication, I/O, and derived type routines?
>>>>>> 
>>>>>> Is it correct, that we keep the number of
>>>>>> - request handles in request arrays,
>>>>>> - ranks an size in/of communicator and group handles,
>>>>>> will stay int?
>>>>>> 
>>>>>> How do we handle disp in 1-sided communication?
>>>>>> 
>>>>>> With this approach, it seems that explicit MPI_Count versions of
>>>>>> a subroutine is needed only for
>>>>>> MPI_[I][NEIGHBOR_]{ALLTOALL|ALLGATHER}{V|W}
>>>>>> MPI_[I]{GATHER|SCATTER}V
>>>>>> MPI_[I]REDUCE_SCATTER
>>>>>> (18 routines)
>>>>>> MPI_WIN_SHARED_QUERY --> *disp
>>>>>> (1 Routine)
>>>>>> MPI_[UN]PACK
>>>>>> MPI_PACK_SIZE _x/_l version should have MPI_Aint (same as for
>>>>>> MPI_...PACK..._external)
>>>>>> (3 routines)
>>>>>> MPI_GET_COUNT
>>>>>> MPI_GET_ELEMENTS (has already an _X version)
>>>>>> MPI_TYPE_SIZE (has already an _X version)
>>>>>> 
>>>>>> Which solution exists for MPI_CONVERSION_FN_NULL?
>>>>>> 
>>>>>> All other 100 routines with "int count" and "int size" seems
>>>>>> to be solved with directly changing it into "MPI_Count count" and "MPI_Count
>>>>>> size".
>>>>>> 
>>>>>> Best regards
>>>>>> Rolf
>>>>>> 
>>>>>> ----- Original Message -----
>>>>>>> From: "mpiwg-large-counts" <mpiwg-large-counts at lists.mpi-forum.org>
>>>>>>> To: "mpiwg-large-counts" <mpiwg-large-counts at lists.mpi-forum.org>
>>>>>>> Cc: "Jeff Squyres" <jsquyres at cisco.com>
>>>>>>> Sent: Saturday, October 19, 2019 10:26:23 PM
>>>>>>> Subject: [Mpiwg-large-counts] Pythonizing instructions
>>>>>> 
>>>>>>> As discussed in our last meeting, I've written a first take at instructions for
>>>>>>> Chapter Authors for how to write Pythonized functions.
>>>>>>> 
>>>>>>> These instructions are not yet intended for a wide audience -- they are intended
>>>>>>> solely for the working group.  Let's iterate on them here in the WG and get
>>>>>>> them to a good enough state before sharing them widely:
>>>>>>> 
>>>>>>> https://github.com/mpiwg-large-count/large-count-issues/wiki/pythonizing
>>>>>>> 
>>>>>>> Puri/Tony/Dan: please try to use these instructions to Pythonize a few bindings.
>>>>>>> The PR for the Pythonization is
>>>>>>> https://github.com/mpiwg-large-count/mpi-standard/pull/2.  All even-numbered
>>>>>>> chapters are done, and a small number of odd-numbered chapters.  Grep for
>>>>>>> `funcdef` in chapter *.tex files to find a chapter that hasn't been Pythonized
>>>>>>> yet (funcdef is the LaTeX macro for hard-coded LaTeX macros).
>>>>>>> 
>>>>>>> Send your feedback here and/or we'll discuss on Wednesday.
>>>>>>> 
>>>>>>> --
>>>>>>> Jeff Squyres
>>>>>>> jsquyres at cisco.com
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> mpiwg-large-counts mailing list
>>>>>>> mpiwg-large-counts at lists.mpi-forum.org
>>>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts
>>>>>> 
>>>>>> --
>>>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de .
>>>>>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 .
>>>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 .
>>>>>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner .
>>>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .
>>>>>> _______________________________________________
>>>>>> mpiwg-large-counts mailing list
>>>>>> mpiwg-large-counts at lists.mpi-forum.org
>>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts
>>>> 
>>>> --
>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de .
>>>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 .
>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 .
>>>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner .
>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .
>>>> _______________________________________________
>>>> mpiwg-large-counts mailing list
>>>> mpiwg-large-counts at lists.mpi-forum.org
>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts
>>> _______________________________________________
>>> mpiwg-large-counts mailing list
>>> mpiwg-large-counts at lists.mpi-forum.org
>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts
>> 
>> 
>> --
>> Jeff Squyres
>> jsquyres at cisco.com
> 
> -- 
> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de .
> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 .
> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 .
> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner .
> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .