[mpi-21] topo mpi (was Re: Ballot 4 - 0-dim MPI_CART_CREATE and MP I_CART_SUB)

[William Yu]

Hi Rolf,

Just curious. Is this implemented already? Sounds like you have a reference implementation built against different MPIs.

Thanks.

________________ Reply Header ________________
Subject:	[mpi-21] Ballot 4 - 0-dim MPI_CART_CREATE and MPI_CART_SUB
Author:	"Rolf Rabenseifner" <rabenseifner_at_[hidden]>
Date:		January 23rd 2008 5:44 pm

This is a discussion-point for MPI 2.1, Ballot 4.

This is a follow up to:
  Questions about Graph_create, Cart_crate, and Cart_coords 
  in http://www.cs.uiuc.edu/homes/wgropp/projects/parallel/MPI/mpi-errata/index.html
with mail discussion in
  http://www.cs.uiuc.edu/homes/wgropp/projects/parallel/MPI/mpi-errata/discuss/topo/

I'm starting a new 2nd stream for the zero-dim discussion.
-------------------------------

This report shows significant differences of MPI implementations
in the (extreme) case of zero-dimensional topologies!

Therefore clarifications may be necessary.
(Although hopefully never an application may produce 
zero-dimensional Cartesian topologies!) 

If there are no objections, I would produce clarifications
for MPI-2.1 according to the behavior of mpich2
and MPI from IBM (except the rank -767705708, see below). 

MPI-1.1 Sect. 6.5.6, page 187, routine MPI_CART_SUB defines
on lines 38-42:
  If a cartesian topology has been created with MPI_CART_CREATE, 
  the function MPI_CART_SUB can be used to partition the 
  communicator group into subgroups that form lower-dimensional 
  cartesian subgrids, and to build for each subgroup a 
  communicator with the associated subgrid cartesian topology. 
  (This function is closely related to MPI_COMM_SPLIT.)

The text clearly says, that the new communicator must
(1) have a Cartesian topology associated
(2) be lower-dimensional in the case of a subgrid

There is no restriction on the input array remain_dims.
Therefore all MPI implementations (that I tested) 
allow that all entries in remain_dims are "false".

I tested several MPI libraries with creating a subgrid
with MPI_Cart_sub(remain_dims=0) from a 1-dim Cartesian
topology.
I tested the subgrid communicator with MPI_Topo_test.
In the case of MPI_CART, I used MPI_Cartdim_get to retrieve
ndims, and MPI_Cart_get for further details: 
 
- mpich2:    MPI_CART, ndims=0, 
  1.0.3      MPI_Cart_get works, 
             but keeps all OUT arguments unchanged
 
- IBM:       MPI_CART, ndims=0, 
  (on SP)    MPI_Cart_get works, 
             but keeps all OUT arguments unchanged
 
- OpenMPI:   MPI_CART, ndims=1, 
  1.2.4      MPI_Cart_get works 
             and returns dims=1, periods=0, coords=0 independent
             from process or periods in the original comm.
             (may be wrong because (2) is not fulfilled) 

- NEC MP/EX: not MPI_CART           
             (may be wrong because (1) is not fulfilled) 

- Voltaire:  not MPI_CART
  (mpich1)   (may be wrong because (1) is not fulfilled) 

With the implementations that return a correct zero-dim Cartesian  
topology, I tested further usage of this zero-dim communicator:
 
- MPI_Comm_size returns 1 and MPI_Comm_rank returns 0 because
  this communicator is like MPI_COMM_SELF, but with Cartesian
  topology associated.

- MPI_Cart_rank(IN ZeroDimComm, IN coords=0, OUT rank)
  Rationale. A zero-dim communicator has zero coords,
             i.e., this routine should not examine the coords
             input argument. 
  --> mpich2: rank = 0 is returned
              (may be because this is the only existing rank 
              in this communicator, this value may make sense,
              independent of the coord, that should not be 
              analyzed)
  --> IBM:    rank = -767705708 is returned 
              (strange value, not MPI_PROC_NULL, not MPI_UNDEFINED)

- MPI_Cart_coords(IN ZeroDimComm, IN rank=0, OUT coords)
  Rationale. A zero-dim communicator has zero coords,
             i.e., this routine should not return anything
             in the coords output argument. 
  --> mpich2 and IBM: coords is not modified (as expected)

- MPI_Cart_sub(IN ZeroDimComm, IN remain_dims=0, OUT subsubcomm)
  Rationale. A zero-dim communicator has zero dimensions,
             i.e., this routine should not examine remain_dims
             and the returned communicator should be again
             a zero-dim Cartesian communicator.
  --> mpich2 and IBM: subsubcomm is a zero-dim Cartesian 
             communicator (as expected)
 
- MPI_Cart_shift(IN ZeroDimComm, IN direction=0, IN disp=1, 
                 OUT src, OUT dest) 
  Rationale. This call is erroneous because in a zero-dim 
             communicator, the direction=0 does not exist.
  mpich2 and IBM: They detect the error and abort. 

(In OpenMPI, all these calls work as expected on a 
 1-dimensional topology on a "MPI_COMM_SELF") 
 
The last test is not addressed by MPI-1.1:
Is it possible to build a zero-dim Cartesian topology 
directly by calling MPI_Cart_create:

  MPI_Cart_create(IN MPI_COMM_SELF, IN ndims=0, IN dims=1, 
                  IN Periods, IN reorder, OUT ZeroDimComm)

Results: All tested MPI implementations return an error
         and abort. 
         (Same on MPI_COMM_WORLD) 

The latter question was raised by Jesper Traeff.
 
Best regards
Rolf

Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner_at_[hidden]
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

_______________________________________________
mpi-21 mailing list
mpi-21_at_[hidden]
http://lists.cs.uiuc.edu/mailman/listinfo/mpi-21