[mpiwg-rma] Questions about multiple overlapped windows

Tue Dec 9 03:59:51 CST 2014

Hi Jeff and Rajeev,

>My ticket would allow one to use MPI_WIN_ALLOCATE to get both
>intranode shared memory and internode RMA access.

>See the text on lines 38-43 on pg 454 if it helps.
>
>See also ln 19-21 on pg 406.
``However, concurrent communications to distinct, overlapping windows
may lead to undefined results.''[pg 406]

Thank you for the detailed explanation and pointers.

I am developing a program verification tool with memory consistency
models.  As a first step, I try to formalize a part of MPI memory
consistency model, and enable it to handle one window.

>>>Another reason is to allow ordered and unordered semantics.  The
>>>option to control this happens during window construction and cannot
>>>be changed.  If one wants one semantic (unordered) for performance and
>>>another (ordered) for correctness, then one needs two windows, one for
>>>each.

It looks so interesting.  I am wondering if you would give a
pointer/reference to usage of multiple windows for ordered and
unordered semantics.  Thanks.

Best,
Tatsuya

Jeff Hammond <jeff.science at gmail.com> wrote:
>On Mon, Dec 8, 2014 at 12:00 AM, Tatsuya Abe <abetatsuya at gmail.com> wrote:
>> Hi Jeff,
>>
>> Thank you so much for the reply.
>>
>>>A good example is using WIN_ALLOCATE_SHARED to create a shared memory
>>>window, then passing the resulting buffer into WIN_CREATE.  This
>>>enables a window to support both intranode+interprocess load-store and
>>>internode RMA.
>>
>> Doesn't MPI_Win_allocate_shared support internode RMA, i.e., when I
>> create a window by using MPI_Win_allocate_shared, cannot I call
>> MPI_Put/MPI_Get for the window?  I haven't found any description about
>> it in MPI 3.0 yet.
>
>WIN_ALLOCATE_SHARED is only valid on an MPI communicator corresponding
>to a shared memory domain, so no, it does not enable internode RMA
>unless you can do load-store between nodes.
>
>And yes, you can still use RMA function calls to access data in a
>shared memory window.  This is essential, for example, if you want to
>use atomics, since some languages (e.g. C99) do not have a portable
>abstraction for these.
>
>> I am a little confused.  Could you let me ask you more details?
>>
>> 1. Do you mean that another window must be created by MPI_Win_create,
>>    in order to use MPI_Put/MPI_Get for a window created by
>>    MPI_Win_allocate_shared?  Is this why multiple windows are allowed
>>    to be overlapped?
>
>Yes, but only for internode access, as noted above.
>
>> 2. Do you mean that the multiple overlapped windows (from
>>    MPI_Win_allocate_shared and MPI_Win_create) become unnecessary in
>>    this example if your ticket #397 passes?
>
>My ticket would allow one to use MPI_WIN_ALLOCATE to get both
>intranode shared memory and internode RMA access.
>
>Jeff
>
>> Tatsuya
>>
>>
>>
>> Jeff Hammond <jeff.science at gmail.com> wrote:
>>>On Sun, Dec 7, 2014 at 4:57 PM, Tatsuya Abe <abetatsuya at gmail.com> wrote:
>>>> Dear all,
>>>>
>>>> I am interested in memory consistency models, and studying program
>>>> verification with memory consistency models.  I started to read MPI
>>>> 3.0 document (and Dr. Torsten Hoefler's tutorial slides).
>>>> http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf
>>>>
>>>> I am wondering if you could give replies to my questions.
>>>> I have two questions about multiple overlapped windows:
>>>>
>>>> 1. (Assume that I create two overlapped windows.)  What happens if I
>>>>    do MPI_Put through one window and MPI_Get through the other window?
>>>
>>>It depends entirely on the synchronization epochs you use and whether
>>>the memory model provided is UNIFIED or SEPARATE.
>>>
>>>> 2. I am wondering what are multiple overlapped windows for.
>>>
>>>To work around obnoxious shortcomings of MPI RMA.
>>>
>>>> 1. I can find no description about the first question in MPI 3.0
>>>>    document.
>>>>
>>>>    I found Torsten's slides (credit by RMA Working Group, MPI Forum),
>>>>    which explicitly showed relations between operations (Load, Store,
>>>>    Get, Put, and Acc) on pp.82--84.  But, it seems an explanation
>>>>    when exactly one window is created,
>>>>
>>>>    For example, page 83 of the slide seems to claim that Put and
>>>>    Get are allowed when they are not NOVL (non-overlapped)
>>>>    through one window in separate semantics.
>>>>
>>>>    One of my questions is ``are Put and Get allowed when they are
>>>>    not NOVL (non-overlapped) through *two* *overlapped* windows
>>>>    in separate semantics?''  I would like to get such table about
>>>>    multiple windows.
>>>
>>>Within the unified model, one can reason about overlapping windows by
>>>recognizing that the private windows of the overlapping windows are
>>>all the same and thus when one synchronizes the public and private
>>>windows of each window with WIN_SYNC, then all of the public windows
>>>are the same as well.
>>>
>>>> 2. I can find no example about multiple overlapped windows except
>>>>    Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get
>>>>    are allowed when they are not NOVL (non-overlapped) through two
>>>>    overlapped windows in separate semantics.
>>>>
>>>>    I cannot come up with any useful example code.  Why are multiple
>>>>    *and* overlapped windows allowed to be active?
>>>
>>>A good example is using WIN_ALLOCATE_SHARED to create a shared memory
>>>window, then passing the resulting buffer into WIN_CREATE.  This
>>>enables a window to support both intranode+interprocess load-store and
>>>internode RMA.  It is useful if one wants to reproduce the behavior of
>>>e.g. ARMCI in implementing Global Arrays.
>>>
>>>The aforementioned motivation will disappear if my ticket
>>>https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 is passed and
>>>part of MPI-4.
>>>
>>>Another reason is to allow ordered and unordered semantics.  The
>>>option to control this happens during window construction and cannot
>>>be changed.  If one wants one semantic (unordered) for performance and
>>>another (ordered) for correctness, then one needs two windows, one for
>>>each.
>>>
>>>Jeff
>>>
>>>--
>>>Jeff Hammond
>>>jeff.science at gmail.com
>>>http://jeffhammond.github.io/
>
>
>
>-- 
>Jeff Hammond
>jeff.science at gmail.com
>http://jeffhammond.github.io/