[MPI3 Fortran] Nonblocking MPI and Fortran temporary memory modifications

Tue Mar 29 11:32:47 CDT 2011

A few minutes ago, we talked in the MPI Forum plenary about this issue and
decided that we as the MPI standardization body will send
to the Fortran Standardization body an official request to
implement this extension of the ASYNCHRONOUS attribute
as it was planned in this J3 vote
> SV: extend ASYNCHRONOUS - invent new attribute - undefined: 8-2-3
because we very seriously need this extension.
Without it, the combination of any nonblocking MPI routine with
Fortran *will not work*.

Rolf

----- Original Message -----
> From: "Rolf Rabenseifner" <rabenseifner at hlrs.de>
> To: "Craig E Rasmussen" <rasmussn at lanl.gov>
> Cc: "MPI-3 Fortran working group" <mpi3-fortran at lists.mpi-forum.org>, "N.M. Maclaren" <nmm1 at cam.ac.uk>, "Bill Long"
> <longb at cray.com>, "Reinhold Bader" <reinhold.bader at lrz.de>, "Brian E. Smith" <smithbr at us.ibm.com>
> Sent: Tuesday, March 29, 2011 3:34:50 PM
> Subject: Re: Nonblocking MPI and Fortran temporary memory modifications
> Hi all,
> 
> this is an excellent basis for the necessary review (deadline March
> 31, 2011!) of
> TR 29113 (Version N1845)
> ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1845.pdf
> from
> http://www.nag.co.uk/sc22wg5/
> 
> I put Brain Smith from IBM also on the CC list.
> 
> The cited papers can be found at
> http://www.j3-fortran.org/doc/year/09/09-235r2.txt
> and
> http://www.j3-fortran.org/doc/year/09/09-231.txt
> 
> Goal of this "extend ASYNCHRONOUS"
> 
> SV: extend ASYNCHRONOUS - invent new attribute - undefined: 8-2-3
> 
> is that the following code is a correct code
> and allows that the MPI library internally used DMA, the NIC,
> or other methods to communicate parts of the array while
> the user application works on another part of the same array:
> 
> USE mpi_f08
> REAL, ASYNCHRONOUS :: buf(100,100)
> CALL MPI_Irecv(buf(1,1:100),...req,...)
> DO j=1,100
> DO i=2,100
> buf(i,j)=....
> END DO
> END DO
> CALL MPI_Wait(req,...)
> 
> It is important that the compiler is not allowed to translate this
> program (by using temporary memory modifications) into
> 
> USE mpi_f08
> REAL, ASYNCHRONOUS :: buf(100,100), buf_1dim(10000)
> EQUIVALENCE (buf(1,1), buf_1dim(1))
> CALL MPI_Irecv(buf(1,1:100),...req,...)
> tmp(1:100)=buf(1,1:100)
> DO j=1,10000
> buf_1dim(j)=...
> END DO
> buf(1,1:100)=tmp(1:100)
> CALL MPI_Wait(req,...)
> 
> While the MPI library receives buf(1,1:100),
> the numerical part overwrites this part of the array
> as part of a numerical optimization to achieve one long
> loop instead of a 2-loop-nesting.
> 
> The current status of Fortran 2008 + TR 29113 (Version N1845)
> is that a compiler can ignore the ASYNCHRONOUS attribute,
> for example, because it does not implement Fortran asynchronous
> I/O in an asynchronous way, i.e., there is no need
> for the compiler to make any restrictions on optimization.
> 
> The goal of the vote
> 
> SV: extend ASYNCHRONOUS - invent new attribute - undefined: 8-2-3
> 
> was, to extend the meaning of the ASYNCHRONOUS attribute
> that it can be used together with MPI or other asynchronous
> software.
> 
> As one can see in
> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/229/mpi-report-F2008-2011-03-26-changeonly-majorpages.pdf
> Page 25 lines 5-14 in the pdf (page 550 in the original document),
> the ASYNCHRONOUS attribute is the only one that can solve the problem.
> 
> The word "SOLVED" is already based on the "extend ASYNCHRONOUS",
> (i.e., the extended meaning of the Fortran ASYNCHRONOUS attribute)
> and can be used for the whole row.
> 
> With the current state of the TR 29113, the whole line is
> "NOT solved".
> 
> VOLATILE is not an option, because it would eliminate
> any numerical optimization on any code that is used
> in overlapping communication and computation.
> 
> Final implication:
> To solve this problem is absolute necessary for any
> Fortran implementation of the MPI library.
> 
> Additionally to the implementation of the "extend ASYNCHRONOUS"
> in the TR 29113 of Fortran 2008, we plan to write into the MPI-3.0
> standard a sentence like this:
> 
> "A valid implementation of the MPI-3.0 standard requires that
> the Fortran compiler in use has to guarantee that the
> ASYNCHRONOUS attribute guarantees that any MPI nonblocking
> operation can be used on data that is marked as ASYNCHRONOUS.
> Other parts of the same data can be used in overlapping computation
> as long as the application uses the ASYNCHRONOUS attribute
> also in this part of the application software."
> 
> Best regards
> Rolf
> 
> ----- Original Message -----
> > From: "Craig E Rasmussen" <rasmussn at lanl.gov>
> > To: "Craig E Rasmussen" <rasmussn at lanl.gov>
> > Cc: "Rolf Rabenseifner" <rabenseifner at hlrs.de>, "MPI-3 Fortran
> > working group" <mpi3-fortran at lists.mpi-forum.org>,
> > "N.M. Maclaren" <nmm1 at cam.ac.uk>, "Bill Long" <longb at cray.com>,
> > "Reinhold Bader" <reinhold.bader at lrz.de>
> > Sent: Tuesday, March 29, 2011 2:50:47 AM
> > Subject: Re: Nonblocking MPI and Fortran temporary memory
> > modifications
> > I've found the relevant paper in J3 meeting minutes (J3 document
> > 09-235r2). The paper is 09-231 "Answer to MPI Forum regarding MPI
> > asynchronous operations" from J3 meeting 188. The minutes state the
> > following regarding this paper:
> >
> > There was a straw vote taken on what syntax to use to inhibit
> > optimizations for MPI asynchronous operations:
> >
> > Paper 09-231 "Answer to MPI Forum regarding MPI asynchronous
> > operations" [Rasmussen] discussed how to prevent code-motion and
> > copy-in/out:
> >
> > SV: extend ASYNCHRONOUS - invent new attribute - undefined: 8-2-3
> >
> >
> > The vote was 8 for ASYNCHRONOUS and 2 for new syntax with 3
> > abstentions.
> >
> > There was also a report from the INTEROP subcommittee:
> >
> > /INTEROP: no further action will be taken on 09-185r1, 09-189r1,
> > 09-191, 09-231 - they are all input for the Interop TR
> > N1761
> >
> > So according to the minutes N1761 was supposed to take this vote and
> > paper 09-231 into account. However, to my knowledge no text was
> > added
> > to the TR in response to this.
> >
> > This was my bad. I should have realized that just because
> > ASYNCHRONOUS
> > is currently part of the Fortran standard, that this keyword can be
> > used outside of Fortran I/O without further words to that affect in
> > the standard.
> >
> > We can discuss what to do in response to this in the telecon this
> > evening.
> >
> > -craig
> >
> >
> > On Mar 28, 2011, at 4:18 PM, Rasmussen, Craig E wrote:
> >
> > I'll look for the vote in the minutes.
> >
> > -craig
> >
> > On Mar 28, 2011, at 4:12 PM, Rolf Rabenseifner wrote:
> >
> > Craig, is there a protocol, file, ... on
> > http://www.nag.co.uk/sc22wg5/
> > where this vote is protocolled?
> > Rolf
> >
> > ----- Original Message -----
> > From: "Craig E Rasmussen"
> > <rasmussn at lanl.gov<mailto:rasmussn at lanl.gov>>
> > To: "Rolf Rabenseifner"
> > <rabenseifner at hlrs.de<mailto:rabenseifner at hlrs.de>>
> > Cc: "MPI-3 Fortran working group"
> > <mpi3-fortran at lists.mpi-forum.org<mailto:mpi3-fortran at lists.mpi-forum.org>>,
> > "N.M. Maclaren" <nmm1 at cam.ac.uk<mailto:nmm1 at cam.ac.uk>>, "Bill Long"
> > <longb at cray.com<mailto:longb at cray.com>>, "Reinhold Bader"
> > <reinhold.bader at lrz.de<mailto:reinhold.bader at lrz.de>>
> > Sent: Monday, March 28, 2011 6:14:00 PM
> > Subject: Re: Nonblocking MPI and Fortran temporary memory
> > modifications
> > We've had several discussions regarding the use of ASYNCHRONOUS for
> > the usage required by MPI on the J3 committee. Some members thought
> > that ASYNCHRONOUS shouldn't be used apart from I/O. However, a vote
> > was taken and a large majority voted that the ASYNCHRONOUS attribute
> > should be used for this purpose.
> >
> > So the question is not about whether a compiler implements
> > ASYNCHRONOUS I/O but whether it respects the ASYNCHRONOUS semantics.
> > Perhaps we need an interp request to formalize this beyond a simple
> > J3
> > member vote.
> >
> > -craig
> >
> >
> > On Mar 27, 2011, at 3:34 AM, Rolf Rabenseifner wrote:
> >
> > Dear all,
> >
> > Reinhold showed me that the ASYNCHRONOUS attribute may not help
> > because a compiler may implement asynchronous Fortran I/O
> > with blocking I/O and therefore may ignore all ASYNCHRONOUS
> > attributes.
> >
> > Please have a look at
> > https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/229/mpi-report-F2008-2011-03-26-changeonly-majorpages.pdf
> >
> > Pages 24-31 in the pdf (pages 549-556 in the original document)
> > show our knowledge about the nonblocking and datatype (MPI_BOTTOM)
> > problems
> > in the combination of Fortran and MPI.
> >
> > Unfortunately in the taböle on page 550, we have to modify all
> > entries
> > about ASYNCHRONOUS into "NOT solved".
> >
> > Do you see further solutions?
> >
> > Do you see further problems with nonblocking and "MPI_BOTTOM",
> > that are not mentioned in this section and that should be.
> >
> > Is the rest of this section correct?
> > If something is incorrect, it would be important to understand why.
> >
> > I've reworked this section based on many discussions and on my best
> > knowledge - but I'm not sure whether this was enough.
> >
> > Best regards
> > Rolf
> >
> >
> > --
> > Dr. Rolf Rabenseifner . . . . . . . . . .. email
> > rabenseifner at hlrs.de<mailto:rabenseifner at hlrs.de>
> > High Performance Computing Center (HLRS) . phone
> > ++49(0)711/685-65530
> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 /
> > 685-65832
> > Head of Dpmt Parallel Computing . . .
> > www.hlrs.de/people/rabenseifner<http://www.hlrs.de/people/rabenseifner>
> > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)
> >
> > --
> > Dr. Rolf Rabenseifner . . . . . . . . . .. email
> > rabenseifner at hlrs.de<mailto:rabenseifner at hlrs.de>
> > High Performance Computing Center (HLRS) . phone
> > ++49(0)711/685-65530
> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 /
> > 685-65832
> > Head of Dpmt Parallel Computing . . .
> > www.hlrs.de/people/rabenseifner<http://www.hlrs.de/people/rabenseifner>
> > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)
> 
> --
> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
> Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

-- 
Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)