[MPI3 Fortran] [Fwd: Library-based ASYNCHRONOUS I/O and SYNC MEMORY]
Jeff Squyres
jsquyres at cisco.com
Fri Sep 5 12:42:01 CDT 2008
We met in Dublin yesterday and had some questions about this
proposal. I have more complete notes, but I thought I'd send the
questions specific to SYNC MEMORY in a separate mail:
- It looks like SYNC MEMORY must be in pairs (it is not explicitly
"start" and "stop"). Is that true? If so, is the compiler smart
enough to figure out if SYNC MEMORY appears in conditional logic,
perhaps something like:
SYNC MEMORY
call MPI_Irecv(buffer, ....)
...
if (foo) then
...
Call MPI_Wait(req)
SYNC MEMORY
...
else
....
Call MPI_Wait(req)
SYNC MEMORY
...
endif
or would you have to move the SYNC_MEMORY outside of the "if" block?
- Is there a way to associate the prevent-moving mechanism with a
specific buffer? For example, there may be a *large* section of code
between the initiation of the MPI non-blocking communication and the
termination. Disabling *all* code movement within that section would
seem undesirable. For example, can we do something like:
SYNC MEMORY my_buffer
call MPI_Irecv(my_buffer, ....)
...
call MPI_Wait(req)
SYNC MEMORY my_buffer
This would *only* inhibit code movement regarding "my_buffer".
On Sep 4, 2008, at 5:26 PM, Hubert Ritzdorf wrote:
> Hi,
>
> attached the aynchronous proposal as provided by Matthijs van Waveren.
>
> Best regards
>
> Hubert
>
> From: Matthijs van Waveren <m_van_waveren at yahoo.fr>
> Date: September 4, 2008 4:58:52 PM IST
> To: Hubert Ritzdorf <ritzdorf at it.neclab.eu>
> Subject: Library-based ASYNCHRONOUS I/O and SYNC MEMORY
>
>
> To: J3
> From: Aleksandar Donev
> Subject: Library-based ASYNCHRONOUS I/O and SYNC MEMORY
> Date: 2008 July 18
>
> Several important libraries have routines for asynchronous data
> transfer, for example, MPI's non-blocking communication routines. The
> use of these routines is essentially identical to asynchronous I/O
> within Fortran 2003, however, the ASYNCHRONOUS attribute only applies
> to the standard-specified asynchronous I/O. The VOLATILE attribute
> may be used to signal to the compiler that a variable may be modified
> asynchronously by an external mechanism, however, this disables
> all optimizations involving the variable, which is not needed. In
> particular, in good programs the variable will not be
> referenced or
> defined while asynchronous data transfer is possibly occuring (this
> is an explicit restriction in the MPI standard), just as required in
> F2003 for variables involved in async I/O.
>
> It is therefore useful to extend the semantics of the ASYNCHRONOUS
> attribute to also apply to library I/O. This was difficult to do in
> Fortran 2003 because the issues of memory consistency were hard to
> specify. However, the same kind of asynchronous data transfer and
> memory consistency issues occur with coarrays, and we now have the
> segment model and SYNC MEMORY to lean on. I therefore consider this
> proposal to be integration and appropriate for immediate incorporation
> into Fortran 2008, especially given the long-existing demand for it
> from the MPI community.
>
> The proposal here requires very little implementation effort.
> Compilers
> that already implement SYNC MEMORY due to parallelism will already
> inhibit code
> motion across SYNC MEMORY for all variables, and this
> proposal merely gives that as a guarantee to the programmer. Compilers
> that implement (true) asynchronous I/O already need to inhibit motion
> across WAIT statements and SYNC MEMORY can be handled identically.
> Only
> compilers that presently ignore ASYNCHRONOUS and plan to implement
> only
> "single-image" coarrays need to actually do some work. Since compilers
> already have to inhibit code motion across certain procedure calls
> the work involved in doing the same for SYNC MEMORY is small.
>
> Technical specification:
> ==============
>
> I propose that the following modification be made to the F2008
> standard:
> We should explicitly allow a variable with the ASYNCHRONOUS attribute
> to be modified or examined by means external to the processor,
> similarly to VOLATILE variables. If such a variable is modified
> or examined externally during a segment, that variable must not
> be
> referenced or define during that segment. Details are in the edits
> below.
>
> This simple modification solves an existing problem with MPI
> non-blocking transfer, namely, the need to prevent movement of code
> across calls to MPI_Wait. The programmer can use SYNC MEMORY to
> indicate to the compiler that ASYNCHRONOUS variables may be affected
> and therefore old copies in registers should be discarded and new
> values written to memory. This is exactly as for coarrays and TARGETs
> (which may be modified by other images) and also just like SYNC
> MEMORY needs to be put around external synchronization routines such
> as MPI_Barrier (see Note 8.39).
>
> Also note that the ASYNCHRONOUS attribute solves another vexing
> problem
> with MPI non-blocking transfer, namely, that of copy in/out. The
> existing rules we have now specify that if both the dummy and the
> actual have the ASYNCHRONOUS attribute, no copy in/out can
> occur
> because either the dummy has to be assumed-shape or the actual
> has to be simply-contiguous. It would be nice if we could say that
> explicitly in the standard (not proposed here because I do not know
> how to word it).
>
> Straw vote:
> ===============
>
> Note that the use of SYNC MEMORY will lead to lots of code segments
> like this:
>
> SYNC MEMORY
> CALL MPI_Wait(request,...) ! Complete communication
> SYNC MEMORY
>
> or
>
> SYNC MEMORY
> CALL MPI_Barrier(comm) ! If used to synchronize images
> SYNC MEMORY
>
> I think it would be a benefit to programmers to give them syntactic
> sugar to do this without requiring rewriting existing codes or
> writing wrappers. It could be achieved through a SYNC procedure
> attribute, which cannot be combined with PURE and is not a procedure
> characteristic. A call to a procedure with the SYNC attribute implies
> a SYNC MEMORY both before and after the call. I do not
> provide edits
> for this but I hope it can be voted on.
>
> ---------------------
> Examples:
>
> --------
> Example 1:
>
> REAL, ASYNCHRONOUS :: buffer(100)
>
> SYNC MEMORY ! This is not really necessary in practice, it should
> be added
> CALL MPI_IRecv(buffer,...,request,...) ! The dummy buffer has the
> ASYNC attribute
> ... ! Code not involving buffer but buffer may be modified outside
> CALL MPI_Wait(request,...)
> SYNC MEMORY
> WRITE(*,*) buffer
>
> --------
> Example 2:
>
> REAL..., ASYNCHRONOUS :: buffer1, buffer2, ...
>
> CALL PrepareNonBlocking(buffer1, buffer2, ...) ! Build internal
> pointers etc.
> ! This may take some time to initialize, but is done only once
> ! No copy in/out will happen if buffers are simply-contiguous
> ! and the interface has ASYNCHRONOUS on the dummies
> ....
> buffer2=...
> SYNC MEMORY
> CALL BeginNonBlocking() ! Start async transfer
> .... ! Cannot
> reference buffers within this segment
> .... ! This may span across many procedure calls or even scoping units
> CALL WaitNonBlocking()
> SYNC MEMORY
> WRITE(*,*) buffer1
>
> ---------------------
>
> Edits:
>
> [88:p1] Clause 5.3.4 on the ASYNCHRONOUS attribute:
> Add to the end of the first sentence:
> ", or a variable that may be referenced, defined, or become undefined,
> by means not specified by the program."
>
> {Note that I specifically do not want to allow pointer or association
> status to be changed asynchronously, as we do for VOLATILE.}
> [88:p2] Clause 5.3.4. Rewrite para 2:
> The base object of a variable shall have the ASYNCHRONOUS attribute
> in a scoping unit if the variable appears in an executable statement
> or specification expression in that scoping unit and any statement
> of the scoping unit is executed while
> -the variable is a pending I/O storage sequence affector (9.6.2.5), or
> -the variable is
> referenced, defined, or become undefined, by means
> not specified by the program and the base object does not have the
> VOLATILE attribute in that scoping unit
> {Note: The second item may be stronger than we want since it requires
> either VOLATILE or ASYNCHRONOUS to be specified. Does this disallow
> existing threaded programs?}
>
> {Note: These rules are meant to be analogues of the rules in 9.6.4
> for Fortran async I/O.}
> [88:p3+] Add a new paragraph after para 3:
> If a variable has the ASYNCHRONOUS attribute but does not have the
> VOLATILE attribute, then:
> -If it is referenced by means not specified by the program during
> the execution of a segment, then it shall not be defined or become
> undefined in a statement executed during that segment.
> -If it is defined or becomes undefined by means not specified by
> the program during the execution of a segment, then it shall not
> be referenced, defined, or become undefined
> in a statement executed
> during that segment, or become associated with a dummy argument that
> has the VALUE attribute during that segment.
>
> [88:NOTE 5.4] Clause 5.3.4. Add a new sentence before the last
> sentence
> of Note 5.4:
> "The ASYNCHRONOUS attribute should also be used to specify variables
> that are involved in asynchronous data transfer performed by external
> libraries, such as non-blocking communication in certain parallel
> programs."
>
> [88:] Clause 5.3.4. Add a new Note 5.4+:
> "The difference between the VOLATILE and ASYNCHRONOUS attributes is
> that the processor may optimize the execution of a segment assuming
> that all asynchronous data transfer happens due to means specified
> by the program. After a new segment begins, the Fortran processor
> should reload the most recent value of an asynchronous object from
> memory when a value is required. Likewise, when a segment ends,
> the processor should store the
> most recent Fortran definition in
> memory. It is the programmer's responsibility to manage any
> interaction
> with non-Fortran processes and to use SYNC MEMORY to delimit segments
> and thus inform the processor to disable certain optimizations. It is
> also the programmer's responsibility to only reference or define the
> variable in segments in which it is not being defined or referenced
> by non-Fortran processes.
>
> For example:
>
> INTERFACE
> SUBROUTINE MY_WRITE(var, n_bytes, id)
> REAL, INTENT(IN), ASYNCHRONOUS :: var(*)
> INTEGER, INTENT(IN) :: n_bytes
> INTEGER, INTENT(OUT) :: id
> END SUBROUTINE
> SUBROUTINE MY_WAIT(id)
> INTEGER, INTENT(IN) :: id
> END SUBROUTINE
> END INTERFACE
>
> REAL :: buffer(100)
> INTEGER :: id
>
> buffer=... ! Definition
>
> SYNC MEMORY ! Segment boundary
> ! No copy shall occur in this call since buffer is simply-contigous:
> CALL MY_WRITE(buffer,
> SIZE(buffer)*(STORAGE_SIZE(buffer)/8), id)
> ! Statements not referencing or defining buffer
> CALL MY_WAIT(id) ! Complete the asynchronous I/O
> SYNC MEMORY ! Segment boundary
>
> buffer=... ! Definition
>
>
> _______________________________________________
> mpi3-fortran mailing list
> mpi3-fortran at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran
--
Jeff Squyres
Cisco Systems
More information about the mpiwg-fortran
mailing list