[Mpi3-rma] [EXTERNAL] Re: Disp_unit in allocate_shared should be returned by win_shared_query?

[Rolf Rabenseifner]

I would like to see an additional advice to users that tell
that the usual way to use disp_unit as extent of the same
datatype on all processes of a window, i.e., to have the 
identical value on all processes on homogeneous systems.

Without such ann advice, users tend to "misuse" such features.

One may add a rationale that non-identical values may need O(p)
memory to store the disp_uni information on each process,
i.e., on all processes together O(p**2) memory.

Such a hint can be added to the existing advice:

MPI-2.2 p337:1-5:

Advice to users. Common choices for disp_unit are 1 (no scaling), 
and (in C syntax) sizeof(type), for a window that consists 
of an array of elements of type type. The later choice will allow 
one to use array indices in RMA calls, and have those scaled
correctly to byte displacements, even in a heterogeneous 
environment. (End of advice to users.)

I asked earlier for such an advice, but it was lost in the discussion
of the final meaning of the interface.

If the RMA roup can formulate such an advice, still enough general,
I expect this would be helpful.
This is of course true for all routines using disp_unit.

Rolf
 
----- Original Message -----
> From: "Torsten Hoefler" <htor at illinois.edu>
> To: "MPI 3.0 Remote Memory Access working group" <mpi3-rma at lists.mpi-forum.org>
> Sent: Wednesday, July 25, 2012 9:16:50 PM
> Subject: Re: [Mpi3-rma] [EXTERNAL] Re: Disp_unit in allocate_shared should be returned by win_shared_query?
> Hi Jim,
> 
> Bill's argument convinced me, I think we should have the displ_unit in
> the query function because this would support future systems and comes
> at essentially no cost. If the unit is the same on all processes, it
> can
> be stored. This can be determined in O(log(P)) or O(1) time with an
> info
> argument. Compression is trivial.
> 
> > We should try to reach a decision on this issue before the end of
> > the
> > week. I think it would be helpful to hear some other opinions. So
> > far,
> > we have three options to choose from:
> >
> > 1) Require that the displacement unit be the same at all processes
> > for
> > MPI_Win_allocate_shared().
> This would never allow us to run across heterogeneous systems. I don't
> know any heterogeneous system that supports the unified model but GPUs
> are coming :-). I agree to Jim that coherency would be tricky, so this
> hardware is unlikely. We could also always lift this restriction in
> future MPI versions, so I'm not opposed.
> 
> > 2) Allow different displacement units at each process in shared
> > memory
> > windows and allow processes to determine the displacement unit at
> > remote
> > processes via MPI_Win_shared_query(). This is in addition to the
> > MPI-2
> > self-query already possible with
> > MPI_Win_get_attr(MPI_WIN_DISP_UNIT).
> Yes, this is what we have at this point.
> 
> > 3) Allow different displacement units at each process in shared
> > memory
> > windows and provide only the MPI-2 self-query already possible with
> > MPI_Win_get_attr(MPI_WIN_DISP_UNIT). Manual communication would be
> > needed to exchange disp_unit information.
> This would require a wording change.
> 
> > A (very) rough summary of the discussion on this issue:
> >
> >  * Given that non-uniform disp_unit values are currently allowed,
> > processes may need to use disp_unit in address calculations.
> They may check if all processes' disp_unit is the same, but that's
> another complication in the interface.
> 
> >  * Requiring programmers to use the disp_unit at each process in
> >  address
> > calculations diminishes the usefulness of the shared memory
> > programming
> > model. It's also not clear if there is a use case to support the
> > non-uniform disp_unit model.
> Agreed.
> 
> I vote for #2.
> 
> Best,
> Torsten
> 
> --
> ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ -----
> Torsten Hoefler | Performance Modeling and Simulation Lead
> Blue Waters Directorate | University of Illinois (UIUC)
> 1205 W Clark Street | Urbana, IL, 61801
> NCSA Building | +01 (217) 244-7736
> _______________________________________________
> mpi3-rma mailing list
> mpi3-rma at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

-- 
Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)