[Mpi-forum] Question on Cartesian Topologies

Tue Mar 8 05:57:29 CST 2011

Rolf:

Your proposal is basically fine with me. Wording comments below.

Bronis

On Tue, 8 Mar 2011, Rolf Rabenseifner wrote:

> Summary of my proposal:
> -----------------------
>
> I propose to clarify:
> MPI-2.2, page 248, line 33 reads
>
>  If ndims is zero then a zero-dimensional Cartesian
>  topology is created.
>
> but should read
>
>  If ndims > 0 and the product of dims[i] is zero
>  then MPI_COMM_NULL is returned in all processes.

The above sentence is fine.

>  If ndims is zero then and in each process,
>  in comm_cart a communicator is retunred
>  that contains only the process itself
>  and that is accociated with a zero-dimensional
>  Cartesian topology (i.e., comm_cart would
>  be the same if all processes in the group
>  of comm would have called MPI_CART_CREATE with
>  ndims=0 and MPI_COMM_SELF as input communicator).

The above sentence has some typos plus the beginning
of it does not parse properly. Replace with:

   If ndims is zero then a communicator that
   contains only the local process accociated
   with a zero-dimensional Cartesian topology
   is returned in comm_cart in each process
   (i.e., the same comm_cart is returned in
   each process in the group of comm as if the
   process called MPI_CART_CREATE with ndims = 0
   and comm_old = MPI_COMM_SELF).

and

> ----------------------
> Details and reasons:
>
> I agree that we should clarify the standard.
> To be compatible with existing implementations
> (they may handle it as an error), I would propose:
>
> Add in MPI-2.2, page 248, line 34,
> before "If ndims is zero":
>
>  If ndims > 0 and the product of dims[i] is zero
>  then MPI_COMM_NULL is returned in all processes.
>
> This solution is in analogy to the definition of
> MPI_GRAPH_CREATE in the case of nnodes==0,
> see MPI-2.2, page 250, lines 30-31.
> (For MPI_DIST_GRAPH_CREATE(_ADJACENT), we decided
> another solution: Here, "isolated processes" are
> allowed, see MPI-2.2, page 253, lines 30-33, and
> page page 255, lines 15-17.
>
> Additionally, I propose to clarify the next sentence,
> i.e., MPI-2.2, page 248, line 33:
> "If ndims is zero then a zero-dimensional Cartesian
> topology is created."
>
> Relevant is only the text in MPI_CART_CREATE and
> MPI_CART_SUB.
> The idea behind this additions in the standardization of
> MPI-2.2 was, that MPI_CART_SUB had not forbidden
> that all remain_dims[i] are false, and threfore
> implementations had to return zero-dimensional
> single-process-communicators in all processes.
>
> In MPI_CART_CREATE, the outcome is not such clear,
> because the number of processes that return
> such a zero-dimensional topology (i.e. a point),
> - can be ONE (and ann others return MPI_COMM_NULL)
> - or SIZE(comm).
>
> The idea behind the new wording in MPI-2.2 was
> clearly to have it identical to the outcome
> of MPI_COMM_SPLIT, i.e., all process return such a
> zero-dimensional topology.
>
> Therefore, I propose to clarify:
> MPI-2.2, page 248, line 33 reads
>
>  If ndims is zero then a zero-dimensional Cartesian
>  topology is created.
>
> but should read
>
>  If ndims > 0 and the product of dims[i] is zero
>  then MPI_COMM_NULL is returned in all processes.
>  If ndims is zero then and in each process,
>  comm_cart contains only the process itself
>  and comm_cart is accociated with a zero-dimensional
>  Cartesian topology (i.e., the outcme is
>  identical to calling this routine by all processes
>  in the group of comm, but with MPI_COMM_SELF
>  as input communicator).
>
> Related lines to "zero-dimensional" are
> - MPI_CARTDIM_GET, MPI-2.2, page 259, lines 35-37
> - MPI_CART_RANK, MPI-2.2, page 260, lines 45-46
> - MPI_CART_COORDS, MPI-2.2, page 261, lines 19-20
> - MPI_CART_SHIFT, MPI-2.2, page 266, lines 1-4
> - MPI_CART_SUB, MPI-2.2, page 267, lines 1-4
>
> Best regards
> Rolf
>
>
> ----- Original Message -----
>> From: "Bronis R. de Supinski" <bronis at llnl.gov>
>> To: "Main MPI Forum mailing list" <mpi-forum at lists.mpi-forum.org>
>> Sent: Monday, March 7, 2011 4:22:09 PM
>> Subject: Re: [Mpi-forum] Question on Cartesian Topologies
>> Michael:
>>
>> I do not agree with your inference. The values in the
>> dims array are not ranks so you cannot extend rank
>> restrictions to them
>>
>> I can envision algorithms for which providing 0 for a
>> dimension indicates that the communicator should be empty.
>> It would be inefficient for the application to check
>> if any dimension is 0. As the MPICH-2 implementation
>> demonstrates, this choice can be supported. I'll
>> admit that your interpretation is reasonable although
>> I think returning MPI_COMM_NULL is better. I also
>> agree that an error should be returned if any of the
>> dims is negative.
>>
>> Regardless, the issue should not be left to inference.
>> We need to update the standard to clarify this issue.
>>
>> Bronis
>>
>>
>> On Mon, 7 Mar 2011, Michael Raymond wrote:
>>
>>>  While there's nothing specified in the definition of
>>> MPI_Cart_create(), you might be able to infer this from the
>>> definition
>>> of MPI_Cart_rank(). It specifies that a coordinate < 0 or >= the
>>> size
>>> of that dimension is out of range. 0 is >= 0, so it must be out of
>>> range.
>>>
>>> Tobias Hilbrich wrote:
>>>> Hi all,
>>>>
>>>> LANL, LLNL and the University of Dresden are working on a follow up
>>>> tool of the runtime correctness checking tools Marmot and Umpire,
>>>> which is done in cooperation by developers of both tools.
>>>>
>>>> We have some issues with basic correctness checks for integers.
>>>> Were for most of the handles used in MPI calls there are fairly
>>>> clear rules on what is allowed and what not, there are sometimes
>>>> close to no rules for integer arguments. Most of the time common
>>>> sense helps to come up with a reasonable assumption, but for the
>>>> definition of cartesian communicators we are somewhat stuck, maybe
>>>> you can help. (Maybe we missed some lines in the standard or are
>>>> not aware of some base rule)
>>>>
>>>> What is unclear to us is "what is a correct definition for a
>>>> cartesian topology?", we found no clear definition for that. So
>>>> basically we care about the "ndims" and "dims" values in calls like
>>>> MPI_Cart_create. Is it allowed to specify a value of 0 for one of
>>>> the entries in the "dims" array? The standard specifies no
>>>> restrictions in the text following MPI_Cart_create, I am referring
>>>> to MPI-2.2 here. E.g. is the following correct or not:
>>>>
>>>> int dims[3]
>>>> dims[0] = 1;
>>>> dims[1] = 1;
>>>> dims[2] = 0; /*Is this ok/suspicious/erroneous ?*/
>>>> MPI_Cart_create( ... , 3 /*ndims*/, dims , ...);
>>>>
>>>> Our little testing revealed that it causes OpenMPI, LAM-MPI,
>>>> MPICH-1 to crash, whereas MPICH-2 returns MPI_COMM_NULL here.
>>>>
>>>> many thanks, Tobias
>>>>
>>>> P.S.: This is probably not the last os such questions, if this is a
>>>> bad way to ask these we would be glad to get suggestions on how to
>>>> raise them. A tool that validates whether an MPI application
>>>> behaves according to the standard or not, hopefully provides some
>>>> motivates to think about these troublesome questions ;)
>>>>
>>>> --
>>>> Dipl.-Inf. Tobias Hilbrich
>>>> Wissenschaftlicher Mitarbeiter
>>>>
>>>> Technische Universitaet Dresden
>>>> Zentrum fuer Informationsdienste und Hochleistungsrechnen (ZIH)
>>>> (Center for Information Services and High Performance Computing
>>>> (ZIH))
>>>> Interdisziplinäre Anwenderunterstützung und Koordination
>>>> (Interdisciplinary Application Development and Coordination)
>>>> 01062 Dresden
>>>> Tel.: +49 (351) 463-32041
>>>> Fax: +49 (351) 463-37773
>>>> E-Mail: tobias.hilbrich at zih.tu-dresden.de
>>>>
>>>>
>>>> _______________________________________________
>>>> mpi-forum mailing list
>>>> mpi-forum at lists.mpi-forum.org
>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum
>>>
>>> --
>>> Michael A. Raymond
>>> Message Passing Toolkit Team
>>> Silicon Graphics Inc
>>> (651) 683-3434
>>>
>>> _______________________________________________
>>> mpi-forum mailing list
>>> mpi-forum at lists.mpi-forum.org
>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum
>>>
>> _______________________________________________
>> mpi-forum mailing list
>> mpi-forum at lists.mpi-forum.org
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum
>
> -- 
> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
> Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)
>