[mpiwg-rma] [MPI Forum] #437: MPI 3.0 Errata: MPI Active Target synchronization requirements for shared memory, 2 alternatives

Rolf Rabenseifner rabenseifner at hlrs.de
Tue Aug 5 12:21:39 CDT 2014

(Small update below in case 4. It is split to 4AC and 4B.)

Dear Bill,

will you be at the Japan Forum meeting?

in our telecon - provided I understood correctly - you told that
you'll try to make a first proposal to solve the shared memory
problems. Please take this email only as a possible input for 
your proposal, i.e., it is not thought as a discussion base.

For me, it is important to finish this discussion more than two
weeks before the Japan meeting, because it must be part 
of the MPI-3.0 errata and MPI-3.1.

The two-week deadline is already Sep.1, 2014, i.e., 4 weeks from now!

Important sections that may need corrections:


p409:16-17 mentions the first time
"The locally allocated memory can be the
target of load/store accesses by remote processes;"

Here we may mention:
The rules for RMA operation do not apply to these
these remote load/store accesses; additional rules apply, 
see, e.g., Section 11.5.4A [the new one below] and 
11.5.5 [on assertions] on pages ... and ... .  

p410:15-21: May need corrections

p436:37-48: Especially the first sentence - #436

How is this section related to shared Memory.
If there are additional rules for shared memory that
do not apply to all unified memory Windows,
then an additional section on "shared & unified" memory
need to be added.

I recommend to define it in the way 
"a load/store access in one process ... a load/store access in 
another process"
and not in the way of "local" and "remote load/store".
For me, after the MPI_Win_allocate_shared,
there is not really memory portions that are local to 
the processes.

The sentence p436:45-46 
"The order in which data is written is not
specified unless further synchronization is used."

If a process issues several load and stores to the 
same location, they are executed in the given sequence
without further need of synchronization

rank 0   rank 1
x=1      print x
x=2      print x

If two processes access the same location, may it possible
that the three prints may be other than 1-1 1-2 2-2? 
Do we want to keep this undefined?

I would propose to add a rationale like

Rationale. If two processes on a ccNUMA node access the same 
memory location with an intermediate process-to-process
synchronization then the outcome is still undefined
because the first memory operation need not to be 
finished before the second memory operation starts.
A well-defined execution may require the following sequence:
1) the memory access on the first process; 
2) a local memory fence to guarantee that the memory
   operation is finished;
3) the process-to-process synchronization (e.g. send/recv)
   to inform the second process; 
4) a local memory fence on the second process to guarantee
   that subsequent memory operations are issued
   to the memory
5) the memory operation on the second process.
To guarantee the outcome of memory accesses in combination
with the synchronization as described in Section 11.5.4A on
page ... (NEW SECTION, SEE BELOW), the synchronization operation
will issue such local memory fences if needed.
End of rationale.

This rationale makes it understandable, what we discussed
in such many emails about the need of MPI_WIN_SYNC.

The major part are the new synchronization rules:
I propose to put them together in one new section after 11.5.4

11.5.4A Shared Memory Synchronization

In the case of an MPI shared memory window additional rules
apply for synchronizing load and store accesses from several
processes to the same location. A location must be at least a byte.

In the following patterns, locations in a shared memory window
are noted with variable names A, B, C,
loads from such windows are noted with load(...) and stores 
are noted by assignments to these variables.

Patterns with active target communication and MPI_Win_sync:

     process P0          process P1
     Sync-to-P1      --> Sync-from-P0

     Sync-to-P1      --> Sync-from-P0

     Sync-to-P1      --> Sync-from-P0
                         load (C)

     Sync-to-P1      --> Sync-from-P0
can be
1.   MPI_Win_fence   --> MPI_Win_fence   1)
2.   MPI_Win_post    --> MPI_Win_start   2)
3.   MPI_Win_complete--> MPI_Win_wait    3)
4AC. Only for the cases with variables 
       A (i.e., write-read) and 
       C (i.e,, write-write):
     any process sync
       from P0 to P1 --> any process sync
                           from P0 to    4)
4B.  Only for the case with variable
       B (i.e., read-write rule) 
     any process sync
       from P0 to P1 --> any process sync
                           from P0 to    4)
 1) MPI_Win_fence synchronizes in both directions and between
    every process in the process group of the window.
 2) The arrow means that P1 is in the origin group passed 
    to MPI_Win_post in P0, and that P0 is in the target 
    group passed to MPI_Win_start.
    Additional calls to MPI_Win_complete 
    (in P1 after MPI_Win_start) and MPI_Win_wait 
    (in P0 after MPI_Win_post) are need. The location of 
    these calls do not influence the guaranteed outcome rules.
 3) The arrow means that P1 is in the target group passed 
    to MPI_Win_start that corresponds to MPI_Win_complete in P0, 
    and P0 is in the origin group passed 
    to MPI_Win_post that corresponds to MPI_Win_wait in P1.
    Additional calls to MPI_Win_start 
    (in P0 before MPI_Win_complete) and MPI_Win_post 
    (in P1 before MPI_Win_wait) are need. The location of 
    these calls do not influence the guaranteed outcome rules.
 4) The synchronization may be done with methods from
    MPI (e.g. send-->recv) or with other methods.
    The requirements for using MPI_Win_sync (e.g. within
    a passive target epoch which may be provided with
    MPI_Win_lock_all) are not shown in thee pattern  

Patterns with lock/unlock synchronization:

Within passive target communication, two locks L1 and L2 may be
scheduled L1 before L2 or L2 before L1.
In the following patterns, the arrow means that the
lock in P0 was scheduled before the lock in P1.

For the following load/store/sync patterns
     process P0          process P1
     MPI_Win_Unlock  --> MPI_Lock 
                           shared or

       shared or
     MPI_Win_Unlock  --> MPI_Lock

     MPI_Win_Unlock  --> MPI_Lock
                         load (C)

Note that each rank of a window is connected to a separate lock.
In a shared window, these locks are not connected to specific
memory portions of the shared window, i.e., each lock can be used
to protect any portion of a shared memory window.

In the patterns above, it is guaranteed 
 - that the load(A) in P1 loads val_1 (this is the Write-read-rule),
 - that the load(B) in P0 is not affected by the store 
   of val_2 in P1 (read-write-rule), and
 - that the load(C) in P1 loads val_4 (write-write-rule).

I do not understand which rules are guaranteed for 
- MPI_Win_flush (and MPI_Win_flush_all)
- MPI_Win_flush_local (and MPI_Win_flush_local_all)
I hope that somebody understands these calls and can produce the
related patterns. 

About Sect. 11.5.5 Assertions - MPI_MODE_NOSTORE:

I expect that the proposal in #429 is not helpful:
it enlarges the hint about "not updated by stores since 
last synchronization" from local stores to all stores by
the whole process group. The reason for this hint is to prevent 
"cache synchronization".

This cache synchronization is local and therefore the
remote stores do not count.

The new proposal is simple:

p452:1 and p452.9: "stores" --> "local stores"

Additionally, I would add the following sentence:
"In the case of a shared memory window, such local stores 
can be issued to any portion of the shared memory window." 

Reason: Nobody should think that "local" means "only to the
window portion that was defined in the local MPI_WIN_ALLOCATE_SHARED".

Best regards

Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

More information about the mpiwg-rma mailing list