[mpiwg-rma] Short question on the ccNUMA memory reality
Bronis R. de Supinski
bronis at llnl.gov
Tue Aug 5 10:00:15 CDT 2014
Rolf:
Re:
> thank you for your answer.
> I was not precise enough with my question.
> My notations should represent store and load instructions,
> i.e., assembler level, i.e., what's going on hardware level.
The "level" is not really relevant. As Pavan noted, unless you use
atomic instructions, you get no guarantees. With ordinary
loads and stores, your snippets all contain race conditions.
The exact result will depend on the system but as a rule,
you can assume that a program with races provides NO guarantees.
Sure, you can eliminate the values coming from registers but
they can come from the cache and you are not provide any
general atomicity guarantees. Most hardware will provide
bit-level atomicity on ordinary loads and stores but if
you straddle a cache line, you lose those guarantees.
Overall, the issues that arise are more complex than you
want to deal with. That's why we have compilers for threaded
programs. Simply put, shared memory programming without compiler
assistance is not something ordinary programmers should do.
Bronis
> x should be part of an MPI shared memory allocated with
> MPI_Win_allocate_shared, i.e., the assembler on rank 0
> does two stores and the assembler on rank 1 does four loads.
>
> The code was
>> rank 0 rank 1
>> _______ print x
>> x=val_1 print x
>> x=val_2 print x
>> _______ print x
>
> When I understand Bill Long correctly, then the 4 load(x)
> end up in
>> old_val
>> val_2 (because it was going through a very slow network path)
>> val_1
>> val_2
> wheras three enterleaved load instructions on rank 0
> (i.e., in the same execution stream (thread) as the two stores)
> will always see
> old_val
> val_1
> val_2
>
> This is independent of compilers, because I only want to
> look at assembbler level.
>
> Bill Long, are you sure? I would expect that all loads go through the
> 1st Level Cache and as soon as it sees val_2 it should not be
> possible to see with a later issued instruction val_1.
>
> Best regards
> Rolf
>
>
> ----- Original Message -----
>> From: "Bill Long" <longb at cray.com>
>> To: "Rolf Rabenseifner" <rabenseifner at hlrs.de>
>> Cc: "MPI WG Remote Memory Access working group" <mpiwg-rma at lists.mpi-forum.org>
>> Sent: Tuesday, August 5, 2014 3:12:58 PM
>> Subject: Re: Short question on the ccNUMA memory reality
>>
>> Hi Rolf,
>>
>>
>> I assume you are expecting answers from people like Pavan and Bill G
>> for the MPI RMA perspective, and for the Fortran rules from me. (For
>> Fortran, map Rank 0 -> image 1 and Rank 1 -> Image 2).
>>
>> On Aug 5, 2014, at 2:33 AM, Rolf Rabenseifner <rabenseifner at hlrs.de>
>> wrote:
>>
>> > Dear expert on ccNUMA,
>> >
>> > three questions, which hopefully may be trivial:
>> >
>> > 1. Question (sequential consistency on one location):
>> > ------------
>> >
>> > Do I understand correctly that in the following patter
>> > on a shared Memory or a ccNUM shared memory
>>
>> I assume you mean what I think of as distributed memory here.
>> Otherwise, these are questions about OpenMP, and the rank 0 / rank
>> 1 separation does not make sense.
>>
>> >
>> > rank 0 rank 1
>> > print x
>> > x=val_1 print x
>> > x=val_2 print x
>> > print x
>>
>> This depends on whether x is declared in a way that makes it
>> accessible from a remote rank. If not, then the code is illegal. So
>> I’ll assume it is accessible.
>>
>> Since there is no synchronization between rank 0 and rank 1, compiler
>> elimination of the x = val_1 assignment is allowed (and likely
>> expected).
>>
>> Even if the assignment is not eliminated, the print x on rank 1
>> involves a “get” of x from rank 0. Assuming it is declared
>> volatile, to eliminate the “snap to local temp” optimization on rank
>> 1, it is still possible to get the values out of order. For
>> example, the print #2 could get routed through the network via some
>> extra long path, while print #3 could get routed on the fastest
>> path, resulting in the two “get” operations appearing at rank 0 in
>> an unpredictable order.
>>
>> [Note that by the Fortran rules, the code is not legal anyway. If
>> you define a variable on one rank and reference it (the print counts
>> as a reference) on another rank the execution segments containing
>> the definition and reference have to be ordered unless the
>> definition and reference are both explicitly atomic. Segment
>> ordering is done with synchronization statements.]
>>
>>
>> >
>> > the print statements can print only in the following
>> > sequence
>> > - some times the previous value
>> > - some times val_1
>> > - and after some time val_2 and it then stays to print val_2
>> >
>> > and that it can never be that a sequence with val_2 before val_1
>> > can be produced, i.e.,
>> > old_val
>> > val_2
>> > val_1
>> > val_2
>> > is impossible.
>> >
>> > Also other values are impossible, e.g., some bit or byte-mix
>> > from val_1 and val_2.
>>
>>
>> On most hardware, assuming the values are “normal” size, such as
>> 64-bit, you will not get mixed bits. However, if you want to ensure
>> that, use the atomic get and put intrinsics. If the accesses had
>> been properly ordered by synchronizations, then you do not need the
>> atomics to ensure whole values.
>>
>>
>> >
>> > 2. Question:
>> > -----------
>> > What is the largest size that the memory operations are atomic,
>> > i.e., that we do not see a bit or byte-mix from val_1 and val_2?
>> > Is it 1, 4, 8, 16 bytes or can it be a total struct that fits
>> > into a cacheline?
>>
>>
>> Since the types in Fortran are parameterized, the processor supplies
>> a KIND value for the integer and logical types for which atomic
>> operations are supported. Typically kind corresponds to 64 bits,
>> but could be 32 on some architectures.
>>
>>
>> >
>> > 3. Question (about two updates):
>> > -----------
>> >
>> > rank 0 rank 1
>> > x=x_ld
>> > y=yold
>> > ---- necessary synchronizations -----
>> > print x (which shows xold)
>> > print y (which shows yold)
>> > ---- necessary synchronizations -----
>> > x=xnew
>> > y=ynew
>> > print x
>> > print y
>> > after some time
>> > print x
>> > print y
>> >
>> > Possible results are
>> > - xold,yold xold,yold xnew,ynew
>> > - xold,yold xnew,yold xnew,ynew
>> > - xold,yold xold,ynew xnew,ynew
>> > i.e., the y=ynew can arrive at another process
>> > faster than the x=xnew, although the storing
>> > process issues the stores in the sequence
>> > x=xnew, y=ynew.
>> > - xold,yold xnew,ynew xnew,ynew
>> >
>> > The assignments should represent the store instructions,
>> > and not the source code (because the compiler may modify
>> > sequence of instructions compared to the source code)
>> >
>> > Do I understand correctly, that the sequence of two
>> > store instructions two two different locations in one process
>> > may be visible at another process in a different sequence?
>>
>> Correct. As before, the “get” operations implied in the print
>> statements could arrive out of order. Also, while unlikely, the
>> compiler could decide to reverse the order of the assignments since
>> they are independent. (Assuming x and y cannot be aliased by, for
>> example, being pointers.)
>>
>> Cheers,
>> Bill
>>
>> >
>> > I ask all these questions to understand which memory model
>> > can be defined for MPI shared memory windows.
>> >
>> > Best regards
>> > Rolf
>> >
>> > --
>> > Dr. Rolf Rabenseifner . . . . . . . . . .. email
>> > rabenseifner at hlrs.de
>> > High Performance Computing Center (HLRS) . phone
>> > ++49(0)711/685-65530
>> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 /
>> > 685-65832
>> > Head of Dpmt Parallel Computing . . .
>> > www.hlrs.de/people/rabenseifner
>> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room
>> > 1.307)
>>
>> Bill Long
>> longb at cray.com
>> Fortran Technical Suport & voice:
>> 651-605-9024
>> Bioinformatics Software Development fax:
>> 651-605-9142
>> Cray Inc./ Cray Plaza, Suite 210/ 380 Jackson St./ St. Paul, MN 55101
>>
>>
>
>
> ----- Original Message -----
>> From: "William Gropp" <wgropp at illinois.edu>
>> To: "MPI WG Remote Memory Access working group" <mpiwg-rma at lists.mpi-forum.org>
>> Cc: "Bill Long" <longb at cray.com>
>> Sent: Tuesday, August 5, 2014 2:38:07 PM
>> Subject: Re: [mpiwg-rma] Short question on the ccNUMA memory reality
>>
>>
>> Question 1 is hard to answer. First, as it came out in my output,
>> the first and last “print x” straddle the two ranks, so I’m not sure
>> what that means. But the following is valid for the compiler:
>>
>>
>> On rank 0. Since x = val_2 follows x = val_1 with no intervening use
>> of x, a good optimizing compiler may eliminate the store to x with
>> val_1 as unnecessary.
>>
>>
>> On rank 1, Since x is never assigned in this block, a compiler is
>> permitted to make a copy in a register (which would have the value x
>> had before this block was entered) and than print that value each
>> time print is called (e.g., it could build the call stack in
>> memory, and just pass that unchanged call stack to print for each
>> call).
>>
>>
>> And don’t forget - there’s a difference between when the print
>> completes on a process/thread and when the output appears - the
>> system could flush all print output from rank 1 before any print
>> output from rank 0 appears.
>>
>>
>> If x is volatile, then the compiler isn’t free to do the second (save
>> x and never load it again).
>>
>>
>> Bill
>>
>>
>>
>>
>>
>>
>> William Gropp
>> Director, Parallel Computing Institute Thomas M. Siebel Chair in
>> Computer Science
>>
>>
>> University of Illinois Urbana-Champaign
>>
>>
>>
>>
>>
>>
>>
>> On Aug 5, 2014, at 2:53 AM, Balaji, Pavan < balaji at anl.gov > wrote:
>>
>>
>> Huh?
>>
>>
>>
>> 1. Question (sequential consistency on one location):
>> ------------
>>
>> Do I understand correctly that in the following patter
>> on a shared Memory or a ccNUM shared memory
>>
>> rank 0 rank 1
>> print x
>> x=val_1 print x
>> x=val_2 print x
>> print x
>>
>> the print statements can print only in the following
>> sequence
>> - some times the previous value
>> - some times val_1
>> - and after some time val_2 and it then stays to print val_2
>>
>> and that it can never be that a sequence with val_2 before val_1
>> can be produced, i.e.,
>> old_val
>> val_2
>> val_1
>> val_2
>> is impossible.
>>
>>
>>
>> _______________________________________________
>> mpiwg-rma mailing list
>> mpiwg-rma at lists.mpi-forum.org
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma
>
> --
> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)
> _______________________________________________
> mpiwg-rma mailing list
> mpiwg-rma at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma
More information about the mpiwg-rma
mailing list