[MPIWG Fortran] Data type of F08 subarray

Wed May 14 02:14:35 CDT 2014

I want to comment on
> > > > That is nasty. Then I will have two data types. I even can not
> > > > assume any relationship between the two types. I have to
> > > > allocate a scratch buffer for the virtual contiguous array in
> > > > MPI_ISEND etc, do memory copying and then free the buffer in
> > > > MPI_WAIT. I'm not sure one can implement it efficiently.

The reason for that interface is very simple:
For blocking calls, the combination of strided arrays
and complicated derived datatypes (eg. produced with type_vector)
was ever allowed for blocking calls. Therefore,
the extension to nonblocking calls is defined with
exactly the same meaning as for blocking calls.
You may name this nasty. Sure. But it would have been
more nasty if we would have defined that the meaning of 
datatype handles should be different for blocking 
and nonblocking calls.

Rolf

----- Original Message -----
> From: "Junchao Zhang" <jczhang at mcs.anl.gov>
> To: "MPI-WG Fortran working group" <mpiwg-fortran at lists.mpi-forum.org>
> Sent: Wednesday, May 14, 2014 12:11:30 AM
> Subject: Re: [MPIWG Fortran] Data type of F08 subarray
> 
> 
> 
> 
> 
> 
> On Tue, May 13, 2014 at 3:21 PM, William Gropp < wgropp at illinois.edu
> > wrote:
> 
> 
> 
> You can always create a new MPI datatype that is the composition of
> the array section and the MPI datatype. For a vector of a simple
> (vector) section, for example, the new datatype simply has the
> product of the strides. Other types are more complex but always
> possible.
> 
> 
> OK. If MPI datatype is represented in a hierarchical tree, then one
> need to combine two MPI datatype trees, which is complicated in
> general.
> In my view, if a user wants to have a complex derived datatype, he
> should create it explicitly with MPI datatype calls, instead of
> doing it implicitly with "subarray X datatype", since that makes
> code hard to understand. MPI standard is better not to support that.
> 
> 
> 
> 
> 
> 
> Bill
> 
> 
> 
> 
> 
> 
> William Gropp
> Director, Parallel Computing Institute Thomas M. Siebel Chair in
> Computer Science
> 
> 
> University of Illinois Urbana-Champaign
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On May 13, 2014, at 3:02 PM, Junchao Zhang wrote:
> 
> 
> 
> 
> 
> 
> On Tue, May 13, 2014 at 2:56 PM, Bill Long < longb at cray.com > wrote:
> 
> 
> 
> 
> On May 13, 2014, at 2:48 PM, Junchao Zhang < jczhang at mcs.anl.gov >
> wrote:
> 
> > 
> > On Tue, May 13, 2014 at 2:37 PM, Bill Long < longb at cray.com >
> > wrote:
> > 
> > On May 13, 2014, at 2:19 PM, Junchao Zhang < jczhang at mcs.anl.gov >
> > wrote:
> > 
> > > 
> > > On Tue, May 13, 2014 at 2:00 PM, Bill Long < longb at cray.com >
> > > wrote:
> > > 
> > > On May 13, 2014, at 12:44 PM, Junchao Zhang < jczhang at mcs.anl.gov
> > > > wrote:
> > > 
> > > > 
> > > > On Tue, May 13, 2014 at 11:56 AM, Rolf Rabenseifner <
> > > > rabenseifner at hlrs.de > wrote:
> > > > > REAL s(100)
> > > > > MPI_SEND(s (1:100:5) , 3, dtype, ...)
> > > > > dtype can only be MPI_REAL. In other words, dtype is kind of
> > > > > redundant here since the type map is actually specified by
> > > > > the
> > > > > subarray.
> > > 
> > > Right. The descriptor for the first argument has a member whose
> > > value is a type code. In principal the library could verify this
> > > is compatible with the data type handle supplied as the third
> > > argument, and issue an error if not. Perhaps in a “debug” mode.
> > > 
> > > > 
> > > > No, if dtype is a vector then it is applied to a virtual
> > > > contiguous array that consists of s(1), s(6), s(11) …
> > > 
> > > dtype is not a vector, is it? That argument is a scalar of type
> > > TYPE(MPI_DATATYPE). At least that is what the interface says.
> > > 
> > > Rolf meant dtype is an MPI datatype created by MPI_Type_vector.
> > > For this case, I will have two datatypes, one from the
> > > MPI_Datatype argument, the other from the choice buffer itself.
> > > It is hard to implement that. Perhaps it is useless since it
> > > obscures the program.
> > 
> > OK. But one of the virtues of the new interface for users is that
> > you do not have to create such data types anymore for array
> > sections. Even if someone did do this, you can detect that the
> > incoming data type is user-created, and in that case ignore the
> > type code in the descriptor. If the program is valid at all, the
> > element length, strides, and extents in the descriptor should be
> > correct.
> > 
> > Yes, I can do that. The hard part is when the subarray is
> > non-contiguous, and it is a non-blocking call. I need to allocate
> > a scratch buffer and pack the subarray. Since it is non-blocking,
> > I can not free the buffer.
> 
> Can you create, locally, a datatype that describes the layout of the
> array section, and then call MPI_Isend again with that data type?
> That avoids the contiguous local buffer and the problem of when to
> free it.
> 
> 
> 
> That is my first thought. But then I realized I have to assume the
> MPI_Datatype argument is for subarray elements.
> 
> 
> 
> Cheers,
> Bill
> 
> 
> 
> 
> 
> 
> > 
> > 
> > Cheers,
> > Bill
> > 
> > 
> > 
> > > 
> > > 
> > > 
> > > Cheers,
> > > Bill
> > > 
> > > 
> > > > 
> > > > That is nasty. Then I will have two data types. I even can not
> > > > assume any relationship between the two types. I have to
> > > > allocate a scratch buffer for the virtual contiguous array in
> > > > MPI_ISEND etc, do memory copying and then free the buffer in
> > > > MPI_WAIT. I'm not sure one can implement it efficiently.
> > > > 
> > > > 
> > > > 
> > > > ----- Original Message -----
> > > > > From: "Junchao Zhang" < jczhang at mcs.anl.gov >
> > > > > To: "MPI-WG Fortran working group" <
> > > > > mpiwg-fortran at lists.mpi-forum.org >
> > > > > Sent: Tuesday, May 13, 2014 6:23:08 PM
> > > > > Subject: Re: [MPIWG Fortran] Data type of F08 subarray
> > > > > 
> > > > > 
> > > > > 
> > > > > Thanks, Rolf. And I feel there is a jump from contiguous
> > > > > subarray to
> > > > > non-contiguous subarray.
> > > > > 
> > > > > 
> > > > > 
> > > > > For contiguous subarray, such as
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > REAL s(100)
> > > > > 
> > > > > MPI_SEND(s(2:50), 3, dtype, ...) s only gives the start
> > > > > address.
> > > > > dtype can be anything, e.g., either a basic type or a derived
> > > > > type
> > > > > created by MPI_Type_vector() etc.
> > > > > 
> > > > > 
> > > > > 
> > > > > For non-contiguous subarray, such as
> > > > > 
> > > > > 
> > > > > 
> > > > > REAL s(100)
> > > > > MPI_SEND(s (1:100:5) , 3, dtype, ...)
> > > > > dtype can only be MPI_REAL. In other words, dtype is kind of
> > > > > redundant here since the type map is actually specified by
> > > > > the
> > > > > subarray.
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > --Junchao Zhang
> > > > > 
> > > > > 
> > > > > On Tue, May 13, 2014 at 10:20 AM, Rolf Rabenseifner <
> > > > > rabenseifner at hlrs.de > wrote:
> > > > > 
> > > > > 
> > > > > Dear Junchao,
> > > > > 
> > > > > MPI-3.0 p25:7-8 describes only communication with language
> > > > > type
> > > > > of the buffer argument matches to the MPI datatype used
> > > > > in the datatype argument.
> > > > > Same p83:36-37.
> > > > > 
> > > > > Therefore, the answer is no and the compiler cannot detect
> > > > > a mismatch beteen language buffer specification and
> > > > > MPI datatype specification.
> > > > > 
> > > > > I hope my answer could help.
> > > > > 
> > > > > Best regards
> > > > > Rolf
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > ----- Original Message -----
> > > > > > From: "Junchao Zhang" < jczhang at mcs.anl.gov >
> > > > > > To: "MPI-WG Fortran working group" <
> > > > > > mpiwg-fortran at lists.mpi-forum.org >
> > > > > > Sent: Tuesday, May 13, 2014 5:08:30 PM
> > > > > > Subject: [MPIWG Fortran] Data type of F08 subarray
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > p626 of MPI-3.0 gives such an example
> > > > > > 
> > > > > > 
> > > > > > REAL s(100), r(100)
> > > > > > CALL MPI_Isend(s(1:100:5), 3, MPI_REAL, ..., rq, ierror)
> > > > > > 
> > > > > > All nonblocking MPI functions behave as if the
> > > > > > user-specified
> > > > > > elements of choice buffers are copied to a contiguous
> > > > > > scratch
> > > > > > buffer
> > > > > > in the MPI runtime environment. All datatype descriptions
> > > > > > (in the
> > > > > > example above, “3, MPI_REAL”) read and store data from and
> > > > > > to this
> > > > > > virtual contiguous scratch buffer ...
> > > > > > 
> > > > > > Here, data type of s(100) match with MPI_REAL, so
> > > > > > everything is
> > > > > > fine.
> > > > > > But I want to know if MPI permits mismatched types, for
> > > > > > example,
> > > > > > can
> > > > > > s(100) be an integer array? If the answer is no, then
> > > > > > compilers can
> > > > > > not detect this error ; if yes, then it is hard to
> > > > > > implement that.
> > > > > > To avoid memory copying to a scratch buffer, I want to use
> > > > > > MPI
> > > > > > datatypes. But if I have two types, one is given by the
> > > > > > choice
> > > > > > buffer itself, the other is given by the MPI_Datatype
> > > > > > argument, how
> > > > > > could I do that?
> > > > > > 
> > > > > > Any thoughts?
> > > > > > 
> > > > > > Thanks
> > > > > > 
> > > > > > --Junchao Zhang
> > > > > > _______________________________________________
> > > > > > mpiwg-fortran mailing list
> > > > > > mpiwg-fortran at lists.mpi-forum.org
> > > > > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> > > > > 
> > > > > --
> > > > > Dr. Rolf Rabenseifner . . . . . . . . . .. email
> > > > > rabenseifner at hlrs.de
> > > > > High Performance Computing Center (HLRS) . phone
> > > > > ++49(0)711/685-65530
> > > > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 /
> > > > > 685-65832
> > > > > Head of Dpmt Parallel Computing . . .
> > > > > www.hlrs.de/people/rabenseifner
> > > > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office:
> > > > > Room 1.307)
> > > > > _______________________________________________
> > > > > mpiwg-fortran mailing list
> > > > > mpiwg-fortran at lists.mpi-forum.org
> > > > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> > > > > 
> > > > > _______________________________________________
> > > > > mpiwg-fortran mailing list
> > > > > mpiwg-fortran at lists.mpi-forum.org
> > > > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> > > > 
> > > > --
> > > > Dr. Rolf Rabenseifner . . . . . . . . . .. email
> > > > rabenseifner at hlrs.de
> > > > High Performance Computing Center (HLRS) . phone
> > > > ++49(0)711/685-65530
> > > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 /
> > > > 685-65832
> > > > Head of Dpmt Parallel Computing . . .
> > > > www.hlrs.de/people/rabenseifner
> > > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room
> > > > 1.307)
> > > > _______________________________________________
> > > > mpiwg-fortran mailing list
> > > > mpiwg-fortran at lists.mpi-forum.org
> > > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> > > > 
> > > > _______________________________________________
> > > > mpiwg-fortran mailing list
> > > > mpiwg-fortran at lists.mpi-forum.org
> > > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> > > 
> > > Bill Long longb at cray.com
> > > Fortran Technical Suport & voice: 651-605-9024
> > > Bioinformatics Software Development fax: 651-605-9142
> > > Cray Inc./ Cray Plaza, Suite 210/ 380 Jackson St./ St. Paul, MN
> > > 55101
> > > 
> > > 
> > > _______________________________________________
> > > mpiwg-fortran mailing list
> > > mpiwg-fortran at lists.mpi-forum.org
> > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> > > 
> > > _______________________________________________
> > > mpiwg-fortran mailing list
> > > mpiwg-fortran at lists.mpi-forum.org
> > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> > 
> > Bill Long longb at cray.com
> > Fortran Technical Suport & voice: 651-605-9024
> > Bioinformatics Software Development fax: 651-605-9142
> > Cray Inc./ Cray Plaza, Suite 210/ 380 Jackson St./ St. Paul, MN
> > 55101
> > 
> > 
> > _______________________________________________
> > mpiwg-fortran mailing list
> > mpiwg-fortran at lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> > 
> > _______________________________________________
> > mpiwg-fortran mailing list
> > mpiwg-fortran at lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> 
> Bill Long longb at cray.com
> Fortran Technical Suport & voice: 651-605-9024
> Bioinformatics Software Development fax: 651-605-9142
> Cray Inc./ Cray Plaza, Suite 210/ 380 Jackson St./ St. Paul, MN 55101
> 
> 
> _______________________________________________
> mpiwg-fortran mailing list
> mpiwg-fortran at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> 
> _______________________________________________
> mpiwg-fortran mailing list
> mpiwg-fortran at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> 
> _______________________________________________
> mpiwg-fortran mailing list
> mpiwg-fortran at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran
> 
> 
> _______________________________________________
> mpiwg-fortran mailing list
> mpiwg-fortran at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-fortran

-- 
Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)