[mpi-21] topo mpi (was Re: Ballot 4 - 0-dim MPI_CART_CREATE and MP I_CART_SUB)

Thu Jan 24 02:20:29 CST 2008

William,

yes, I reported the bahavior of 5 different MPI existing MPI 
implementations. And I attached my test program that calls
the MPI topology functions and reports the behavior of the
MPI library to which it is compiled and linked.

Using this test program, everybody can detect that different
MPI implementations are giving quite different answers
and therefore, there is a need for clarifying the MPI 1.1
standard.

Best regards
Rolf

On Thu, 24 Jan 2008 05:02:47 +0800
 "William Yu" <wyu_at_[hidden]> wrote:
>Hi Rolf,
>
>Just curious. Is this implemented already? Sounds like you have a reference implementation built against different MPIs.
>
>Thanks.
>
>________________ Reply Header ________________
>Subject:	[mpi-21] Ballot 4 - 0-dim MPI_CART_CREATE and MPI_CART_SUB
>Author:	"Rolf Rabenseifner" <rabenseifner_at_[hidden]>
>Date:		January 23rd 2008 5:44 pm
>
>This is a discussion-point for MPI 2.1, Ballot 4.
>
>This is a follow up to:
>  Questions about Graph_create, Cart_crate, and Cart_coords 
>  in http://www.cs.uiuc.edu/homes/wgropp/projects/parallel/MPI/mpi-errata/index.html
>with mail discussion in
>  http://www.cs.uiuc.edu/homes/wgropp/projects/parallel/MPI/mpi-errata/discuss/topo/
>
>I'm starting a new 2nd stream for the zero-dim discussion.
>-------------------------------
>
>This report shows significant differences of MPI implementations
>in the (extreme) case of zero-dimensional topologies!
>
>Therefore clarifications may be necessary.
>(Although hopefully never an application may produce 
>zero-dimensional Cartesian topologies!) 
>
>If there are no objections, I would produce clarifications
>for MPI-2.1 according to the behavior of mpich2
>and MPI from IBM (except the rank -767705708, see below). 
>
>MPI-1.1 Sect. 6.5.6, page 187, routine MPI_CART_SUB defines
>on lines 38-42:
>  If a cartesian topology has been created with MPI_CART_CREATE, 
>  the function MPI_CART_SUB can be used to partition the 
>  communicator group into subgroups that form lower-dimensional 
>  cartesian subgrids, and to build for each subgroup a 
>  communicator with the associated subgrid cartesian topology. 
>  (This function is closely related to MPI_COMM_SPLIT.)
>
>The text clearly says, that the new communicator must
>(1) have a Cartesian topology associated
>(2) be lower-dimensional in the case of a subgrid
>
>There is no restriction on the input array remain_dims.
>Therefore all MPI implementations (that I tested) 
>allow that all entries in remain_dims are "false".
>
>I tested several MPI libraries with creating a subgrid
>with MPI_Cart_sub(remain_dims=0) from a 1-dim Cartesian
>topology.
>I tested the subgrid communicator with MPI_Topo_test.
>In the case of MPI_CART, I used MPI_Cartdim_get to retrieve
>ndims, and MPI_Cart_get for further details: 
> 
>- mpich2:    MPI_CART, ndims=0, 
>  1.0.3      MPI_Cart_get works, 
>             but keeps all OUT arguments unchanged
> 
>- IBM:       MPI_CART, ndims=0, 
>  (on SP)    MPI_Cart_get works, 
>             but keeps all OUT arguments unchanged
> 
>- OpenMPI:   MPI_CART, ndims=1, 
>  1.2.4      MPI_Cart_get works 
>             and returns dims=1, periods=0, coords=0 independent
>             from process or periods in the original comm.
>             (may be wrong because (2) is not fulfilled) 
>
>- NEC MP/EX: not MPI_CART           
>             (may be wrong because (1) is not fulfilled) 
>
>- Voltaire:  not MPI_CART
>  (mpich1)   (may be wrong because (1) is not fulfilled) 
>
>With the implementations that return a correct zero-dim Cartesian  
>topology, I tested further usage of this zero-dim communicator:
> 
>- MPI_Comm_size returns 1 and MPI_Comm_rank returns 0 because
>  this communicator is like MPI_COMM_SELF, but with Cartesian
>  topology associated.
>
>- MPI_Cart_rank(IN ZeroDimComm, IN coords=0, OUT rank)
>  Rationale. A zero-dim communicator has zero coords,
>             i.e., this routine should not examine the coords
>             input argument. 
>  --> mpich2: rank = 0 is returned
>              (may be because this is the only existing rank 
>              in this communicator, this value may make sense,
>              independent of the coord, that should not be 
>              analyzed)
>  --> IBM:    rank = -767705708 is returned 
>              (strange value, not MPI_PROC_NULL, not MPI_UNDEFINED)
>
>- MPI_Cart_coords(IN ZeroDimComm, IN rank=0, OUT coords)
>  Rationale. A zero-dim communicator has zero coords,
>             i.e., this routine should not return anything
>             in the coords output argument. 
>  --> mpich2 and IBM: coords is not modified (as expected)
>
>- MPI_Cart_sub(IN ZeroDimComm, IN remain_dims=0, OUT subsubcomm)
>  Rationale. A zero-dim communicator has zero dimensions,
>             i.e., this routine should not examine remain_dims
>             and the returned communicator should be again
>             a zero-dim Cartesian communicator.
>  --> mpich2 and IBM: subsubcomm is a zero-dim Cartesian 
>             communicator (as expected)
> 
>- MPI_Cart_shift(IN ZeroDimComm, IN direction=0, IN disp=1, 
>                 OUT src, OUT dest) 
>  Rationale. This call is erroneous because in a zero-dim 
>             communicator, the direction=0 does not exist.
>  mpich2 and IBM: They detect the error and abort. 
>
>(In OpenMPI, all these calls work as expected on a 
> 1-dimensional topology on a "MPI_COMM_SELF") 
> 
>The last test is not addressed by MPI-1.1:
>Is it possible to build a zero-dim Cartesian topology 
>directly by calling MPI_Cart_create:
>
>  MPI_Cart_create(IN MPI_COMM_SELF, IN ndims=0, IN dims=1, 
>                  IN Periods, IN reorder, OUT ZeroDimComm)
>
>Results: All tested MPI implementations return an error
>         and abort. 
>         (Same on MPI_COMM_WORLD) 
>
>The latter question was raised by Jesper Traeff.
> 
>Best regards
>Rolf
>
>
>
>Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner_at_[hidden]
>High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
>University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
>Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
>Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)
>
>
>_______________________________________________
>mpi-21 mailing list
>mpi-21_at_[hidden]
>http://lists.cs.uiuc.edu/mailman/listinfo/mpi-21
>
>
>
>_______________________________________________
>mpi-21 mailing list
>mpi-21_at_[hidden]
>http://lists.cs.uiuc.edu/mailman/listinfo/mpi-21

Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner_at_[hidden]
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)