[Mpi-forum] Significant discussion point with Ticket 269

[Fab Tillier]

Adam T. Moody wrote on Thu, 17 Mar 2011 at 12:43:49

> Fab Tillier wrote:
> 
>> Hi Adam,
>> 
>> -----Original Message----- From: mpi-forum-bounces at lists.mpi-forum.org
>> [mailto:mpi-forum- bounces at lists.mpi-forum.org] On Behalf Of Adam T.
>> Moody Sent: Wednesday, March 16, 2011 5:47 PM
>> 
>>> The displacements in the new w functions are not semantically the same
>>> as in Alltoallw. See p 146, lines 41-44 vs p 169, line 26.  In
>>> alltoallw, the displacements are given in number of bytes from the
>>> start of the buffer, but for the new functions, they are given as an
>>> integer number of extents of the corresponding datatype.
>> 
>> The displacement parameters are identical in definition to those of the 'V'
>> variants of those functions, only the datatype (recvtype and/or sendtype)
>> was turned to an array and pluralized.  I had missed the difference between
>> MPI_Alltoallv and MPI_Alltoallw, thanks for catching that.
>> 
>>> First, we should be consistent with alltoallw or otherwise use a
>>> suffix other than 'w'.
>> 
>> I don't know that we can succeed in achieving our goals of allowing users to
>> use large datatypes if we change to byte counts without going to a different
>> (larger) datatype - an 'int' may not be enough displacement for a single
>> datatype.  If we are to change the suffix and use different parameters, let's
>> not mess around with using integer counts and just go to MPI_Count too, not
>> just the displacements.  If we consider going to MPI_Count for count
>> parameters, we might as well just stick with a MPI_Count version of the 'v'
>> functions and avoid the 'w' all together.
> 
> For the sake of discussion, let's go with 'x', and then like you
> suggest, we could create an 'x' set by extending all of the 'v'
> collectives to take MPI_Count instead of int.  I think that could work.
> 
>>> Second, specifying the displacement in terms of datatype extents
>>> restricts some codes from using these functions.  For example, in
>>> gatherw, what if the root needs to receive a 2-byte datatype starting
>>> at byte position 1 in its receive buffer?  I don't think this can be
>>> done under the current definition.
>> 
>> Can't a user resize the datatype and set a LB so that the received
>> datatype is offset by 1 byte?
>> 
> You're right.  That would work.  The down side is that this may require
> a user to create one datatype for each process, where I'm guessing most
> apps can use the same datatype (or a small set of different datatypes)
> in many cases but only change the displacement.  In this case, it would
> be possible, but it's ugly.

Having the displacement in alltoallw in bytes but specified as an int, especially since it's always relative to sendbuf, is just terrible.  If displacements are going to be integers, I strongly believe they cannot be in units of bytes and still be useful, especially since the main motivation for introducing these new collectives is to allow large counts to be encapsulated in derived datatypes.  The bottom line is that  the inconsistency between alltoallw and the new 'w' functions is a result of alltoallw being poorly defined.  We shouldn't define the new 'w' functions as badly just for consistency's sake.
 
>>> Third, if we decide to define the displacement in bytes, as alltoallw
>>> does, then we should really think about making the displacement
>>> datatype an MPI_Aint, or perhaps even MPI_Count.  Using an int
>>> datatype will lead to other problems.
>> 
>> If we change these new functions to use MPI_Aint, then we should really
>> create a new version of Alltoallw to match.  There are other functions that
>> use the wrong 'type' for parameters so we need to be careful about 'the
>> slippery slope'.
> 
> Yes, I agree.  Of these ideas so far, I'm leaning toward the 'x'
> extension of the 'v' collectives, but I need to think about it more.

My goal with this ticket was to get minimal "complete" functionality for large derived types.  My hope was always that it would be replaced before MPI3 is finalized with something better, but driven by *somebody else*.  If it didn't get replaced, at least the standard would provide full (though cumbersome) functionality.

-Fab

>> Oy, the baggage we have to deal with to preserve back compat...
>> 
>> -Fab
>> 
>>> -Adam
>>> _______________________________________________
>>> mpi-forum mailing list
>>> mpi-forum at lists.mpi-forum.org
>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum
>> _______________________________________________
>> mpi-forum mailing list
>> mpi-forum at lists.mpi-forum.org
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum
> _______________________________________________
> mpi-forum mailing list
> mpi-forum at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum