[mpiwg-rma] Questions about multiple overlapped windows

[Jeff Hammond]

On Tue, Dec 9, 2014 at 1:59 AM, Tatsuya Abe <abetatsuya at gmail.com> wrote:
> Hi Jeff and Rajeev,
>
>>My ticket would allow one to use MPI_WIN_ALLOCATE to get both
>>intranode shared memory and internode RMA access.
>
>>See the text on lines 38-43 on pg 454 if it helps.
>>
>>See also ln 19-21 on pg 406.
> ``However, concurrent communications to distinct, overlapping windows
> may lead to undefined results.''[pg 406]
>
> Thank you for the detailed explanation and pointers.
>
> I am developing a program verification tool with memory consistency
> models.  As a first step, I try to formalize a part of MPI memory
> consistency model, and enable it to handle one window.

I would not worry about overlapping windows.  Anyone using it is on
their own anyways.

>>>>Another reason is to allow ordered and unordered semantics.  The
>>>>option to control this happens during window construction and cannot
>>>>be changed.  If one wants one semantic (unordered) for performance and
>>>>another (ordered) for correctness, then one needs two windows, one for
>>>>each.
>
> It looks so interesting.  I am wondering if you would give a
> pointer/reference to usage of multiple windows for ordered and
> unordered semantics.  Thanks.

There is none.  It is merely something one can infer from the text of
the standard.

Jeff

> Best,
> Tatsuya
>
>
> Jeff Hammond <jeff.science at gmail.com> wrote:
>>On Mon, Dec 8, 2014 at 12:00 AM, Tatsuya Abe <abetatsuya at gmail.com> wrote:
>>> Hi Jeff,
>>>
>>> Thank you so much for the reply.
>>>
>>>>A good example is using WIN_ALLOCATE_SHARED to create a shared memory
>>>>window, then passing the resulting buffer into WIN_CREATE.  This
>>>>enables a window to support both intranode+interprocess load-store and
>>>>internode RMA.
>>>
>>> Doesn't MPI_Win_allocate_shared support internode RMA, i.e., when I
>>> create a window by using MPI_Win_allocate_shared, cannot I call
>>> MPI_Put/MPI_Get for the window?  I haven't found any description about
>>> it in MPI 3.0 yet.
>>
>>WIN_ALLOCATE_SHARED is only valid on an MPI communicator corresponding
>>to a shared memory domain, so no, it does not enable internode RMA
>>unless you can do load-store between nodes.
>>
>>And yes, you can still use RMA function calls to access data in a
>>shared memory window.  This is essential, for example, if you want to
>>use atomics, since some languages (e.g. C99) do not have a portable
>>abstraction for these.
>>
>>> I am a little confused.  Could you let me ask you more details?
>>>
>>> 1. Do you mean that another window must be created by MPI_Win_create,
>>>    in order to use MPI_Put/MPI_Get for a window created by
>>>    MPI_Win_allocate_shared?  Is this why multiple windows are allowed
>>>    to be overlapped?
>>
>>Yes, but only for internode access, as noted above.
>>
>>> 2. Do you mean that the multiple overlapped windows (from
>>>    MPI_Win_allocate_shared and MPI_Win_create) become unnecessary in
>>>    this example if your ticket #397 passes?
>>
>>My ticket would allow one to use MPI_WIN_ALLOCATE to get both
>>intranode shared memory and internode RMA access.
>>
>>Jeff
>>
>>> Tatsuya
>>>
>>>
>>>
>>> Jeff Hammond <jeff.science at gmail.com> wrote:
>>>>On Sun, Dec 7, 2014 at 4:57 PM, Tatsuya Abe <abetatsuya at gmail.com> wrote:
>>>>> Dear all,
>>>>>
>>>>> I am interested in memory consistency models, and studying program
>>>>> verification with memory consistency models.  I started to read MPI
>>>>> 3.0 document (and Dr. Torsten Hoefler's tutorial slides).
>>>>> http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf
>>>>>
>>>>> I am wondering if you could give replies to my questions.
>>>>> I have two questions about multiple overlapped windows:
>>>>>
>>>>> 1. (Assume that I create two overlapped windows.)  What happens if I
>>>>>    do MPI_Put through one window and MPI_Get through the other window?
>>>>
>>>>It depends entirely on the synchronization epochs you use and whether
>>>>the memory model provided is UNIFIED or SEPARATE.
>>>>
>>>>> 2. I am wondering what are multiple overlapped windows for.
>>>>
>>>>To work around obnoxious shortcomings of MPI RMA.
>>>>
>>>>> 1. I can find no description about the first question in MPI 3.0
>>>>>    document.
>>>>>
>>>>>    I found Torsten's slides (credit by RMA Working Group, MPI Forum),
>>>>>    which explicitly showed relations between operations (Load, Store,
>>>>>    Get, Put, and Acc) on pp.82--84.  But, it seems an explanation
>>>>>    when exactly one window is created,
>>>>>
>>>>>    For example, page 83 of the slide seems to claim that Put and
>>>>>    Get are allowed when they are not NOVL (non-overlapped)
>>>>>    through one window in separate semantics.
>>>>>
>>>>>    One of my questions is ``are Put and Get allowed when they are
>>>>>    not NOVL (non-overlapped) through *two* *overlapped* windows
>>>>>    in separate semantics?''  I would like to get such table about
>>>>>    multiple windows.
>>>>
>>>>Within the unified model, one can reason about overlapping windows by
>>>>recognizing that the private windows of the overlapping windows are
>>>>all the same and thus when one synchronizes the public and private
>>>>windows of each window with WIN_SYNC, then all of the public windows
>>>>are the same as well.
>>>>
>>>>> 2. I can find no example about multiple overlapped windows except
>>>>>    Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get
>>>>>    are allowed when they are not NOVL (non-overlapped) through two
>>>>>    overlapped windows in separate semantics.
>>>>>
>>>>>    I cannot come up with any useful example code.  Why are multiple
>>>>>    *and* overlapped windows allowed to be active?
>>>>
>>>>A good example is using WIN_ALLOCATE_SHARED to create a shared memory
>>>>window, then passing the resulting buffer into WIN_CREATE.  This
>>>>enables a window to support both intranode+interprocess load-store and
>>>>internode RMA.  It is useful if one wants to reproduce the behavior of
>>>>e.g. ARMCI in implementing Global Arrays.
>>>>
>>>>The aforementioned motivation will disappear if my ticket
>>>>https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 is passed and
>>>>part of MPI-4.
>>>>
>>>>Another reason is to allow ordered and unordered semantics.  The
>>>>option to control this happens during window construction and cannot
>>>>be changed.  If one wants one semantic (unordered) for performance and
>>>>another (ordered) for correctness, then one needs two windows, one for
>>>>each.
>>>>
>>>>Jeff
>>>>
>>>>--
>>>>Jeff Hammond
>>>>jeff.science at gmail.com
>>>>http://jeffhammond.github.io/
>>
>>
>>
>>--
>>Jeff Hammond
>>jeff.science at gmail.com
>>http://jeffhammond.github.io/



-- 
Jeff Hammond
jeff.science at gmail.com
http://jeffhammond.github.io/