[mpiwg-rma] Questions about multiple overlapped windows

Tue Dec 9 20:47:06 CST 2014

>I would not worry about overlapping windows.  Anyone using it is on
>their own anyways.

>There is none.  It is merely something one can infer from the text of
>the standard.

>This paper might interest you then: http://htor.inf.ethz.ch/publications/index.php?pub=201 . It presents a formal model for MPI-3 RMA.

I'll read the text again, and try the paper.  I'll send an e-mail to the ML if I have any question.
Thank you.

Tatsuya

Rajeev Thakur <thakur at mcs.anl.gov> wrote:
>> I am developing a program verification tool with memory consistency
>> models.  As a first step, I try to formalize a part of MPI memory
>> consistency model, and enable it to handle one window.
>
>This paper might interest you then: http://htor.inf.ethz.ch/publications/index.php?pub=201 . It presents a formal model for MPI-3 RMA.
>
>Rajeev
>
>

>> ``However, concurrent communications to distinct, overlapping windows
>> may lead to undefined results.''[pg 406]
>>
>> Thank you for the detailed explanation and pointers.
>>
>> I am developing a program verification tool with memory consistency
>> models.  As a first step, I try to formalize a part of MPI memory
>> consistency model, and enable it to handle one window.
>
>I would not worry about overlapping windows.  Anyone using it is on
>their own anyways.
>
>>>>>Another reason is to allow ordered and unordered semantics.  The
>>>>>option to control this happens during window construction and cannot
>>>>>be changed.  If one wants one semantic (unordered) for performance and
>>>>>another (ordered) for correctness, then one needs two windows, one for
>>>>>each.
>>
>> It looks so interesting.  I am wondering if you would give a
>> pointer/reference to usage of multiple windows for ordered and
>> unordered semantics.  Thanks.
>
>There is none.  It is merely something one can infer from the text of
>the standard.
>
>Jeff
>
>On Dec 9, 2014, at 3:59 AM, Tatsuya Abe <abetatsuya at gmail.com> wrote:
>
>> Hi Jeff and Rajeev,
>> 
>>> My ticket would allow one to use MPI_WIN_ALLOCATE to get both
>>> intranode shared memory and internode RMA access.
>> 
>>> See the text on lines 38-43 on pg 454 if it helps.
>>> 
>>> See also ln 19-21 on pg 406.
>> ``However, concurrent communications to distinct, overlapping windows
>> may lead to undefined results.''[pg 406]
>> 
>> Thank you for the detailed explanation and pointers.
>> 
>> I am developing a program verification tool with memory consistency
>> models.  As a first step, I try to formalize a part of MPI memory
>> consistency model, and enable it to handle one window.
>> 
>>>>> Another reason is to allow ordered and unordered semantics.  The
>>>>> option to control this happens during window construction and cannot
>>>>> be changed.  If one wants one semantic (unordered) for performance and
>>>>> another (ordered) for correctness, then one needs two windows, one for
>>>>> each.
>> 
>> It looks so interesting.  I am wondering if you would give a
>> pointer/reference to usage of multiple windows for ordered and
>> unordered semantics.  Thanks.
>> 
>> Best,
>> Tatsuya
>> 
>> 
>> Jeff Hammond <jeff.science at gmail.com> wrote:
>>> On Mon, Dec 8, 2014 at 12:00 AM, Tatsuya Abe <abetatsuya at gmail.com> wrote:
>>>> Hi Jeff,
>>>> 
>>>> Thank you so much for the reply.
>>>> 
>>>>> A good example is using WIN_ALLOCATE_SHARED to create a shared memory
>>>>> window, then passing the resulting buffer into WIN_CREATE.  This
>>>>> enables a window to support both intranode+interprocess load-store and
>>>>> internode RMA.
>>>> 
>>>> Doesn't MPI_Win_allocate_shared support internode RMA, i.e., when I
>>>> create a window by using MPI_Win_allocate_shared, cannot I call
>>>> MPI_Put/MPI_Get for the window?  I haven't found any description about
>>>> it in MPI 3.0 yet.
>>> 
>>> WIN_ALLOCATE_SHARED is only valid on an MPI communicator corresponding
>>> to a shared memory domain, so no, it does not enable internode RMA
>>> unless you can do load-store between nodes.
>>> 
>>> And yes, you can still use RMA function calls to access data in a
>>> shared memory window.  This is essential, for example, if you want to
>>> use atomics, since some languages (e.g. C99) do not have a portable
>>> abstraction for these.
>>> 
>>>> I am a little confused.  Could you let me ask you more details?
>>>> 
>>>> 1. Do you mean that another window must be created by MPI_Win_create,
>>>>   in order to use MPI_Put/MPI_Get for a window created by
>>>>   MPI_Win_allocate_shared?  Is this why multiple windows are allowed
>>>>   to be overlapped?
>>> 
>>> Yes, but only for internode access, as noted above.
>>> 
>>>> 2. Do you mean that the multiple overlapped windows (from
>>>>   MPI_Win_allocate_shared and MPI_Win_create) become unnecessary in
>>>>   this example if your ticket #397 passes?
>>> 
>>> My ticket would allow one to use MPI_WIN_ALLOCATE to get both
>>> intranode shared memory and internode RMA access.
>>> 
>>> Jeff
>>> 
>>>> Tatsuya
>>>> 
>>>> 
>>>> 
>>>> Jeff Hammond <jeff.science at gmail.com> wrote:
>>>>> On Sun, Dec 7, 2014 at 4:57 PM, Tatsuya Abe <abetatsuya at gmail.com> wrote:
>>>>>> Dear all,
>>>>>> 
>>>>>> I am interested in memory consistency models, and studying program
>>>>>> verification with memory consistency models.  I started to read MPI
>>>>>> 3.0 document (and Dr. Torsten Hoefler's tutorial slides).
>>>>>> http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf
>>>>>> 
>>>>>> I am wondering if you could give replies to my questions.
>>>>>> I have two questions about multiple overlapped windows:
>>>>>> 
>>>>>> 1. (Assume that I create two overlapped windows.)  What happens if I
>>>>>>   do MPI_Put through one window and MPI_Get through the other window?
>>>>> 
>>>>> It depends entirely on the synchronization epochs you use and whether
>>>>> the memory model provided is UNIFIED or SEPARATE.
>>>>> 
>>>>>> 2. I am wondering what are multiple overlapped windows for.
>>>>> 
>>>>> To work around obnoxious shortcomings of MPI RMA.
>>>>> 
>>>>>> 1. I can find no description about the first question in MPI 3.0
>>>>>>   document.
>>>>>> 
>>>>>>   I found Torsten's slides (credit by RMA Working Group, MPI Forum),
>>>>>>   which explicitly showed relations between operations (Load, Store,
>>>>>>   Get, Put, and Acc) on pp.82--84.  But, it seems an explanation
>>>>>>   when exactly one window is created,
>>>>>> 
>>>>>>   For example, page 83 of the slide seems to claim that Put and
>>>>>>   Get are allowed when they are not NOVL (non-overlapped)
>>>>>>   through one window in separate semantics.
>>>>>> 
>>>>>>   One of my questions is ``are Put and Get allowed when they are
>>>>>>   not NOVL (non-overlapped) through *two* *overlapped* windows
>>>>>>   in separate semantics?''  I would like to get such table about
>>>>>>   multiple windows.
>>>>> 
>>>>> Within the unified model, one can reason about overlapping windows by
>>>>> recognizing that the private windows of the overlapping windows are
>>>>> all the same and thus when one synchronizes the public and private
>>>>> windows of each window with WIN_SYNC, then all of the public windows
>>>>> are the same as well.
>>>>> 
>>>>>> 2. I can find no example about multiple overlapped windows except
>>>>>>   Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get
>>>>>>   are allowed when they are not NOVL (non-overlapped) through two
>>>>>>   overlapped windows in separate semantics.
>>>>>> 
>>>>>>   I cannot come up with any useful example code.  Why are multiple
>>>>>>   *and* overlapped windows allowed to be active?
>>>>> 
>>>>> A good example is using WIN_ALLOCATE_SHARED to create a shared memory
>>>>> window, then passing the resulting buffer into WIN_CREATE.  This
>>>>> enables a window to support both intranode+interprocess load-store and
>>>>> internode RMA.  It is useful if one wants to reproduce the behavior of
>>>>> e.g. ARMCI in implementing Global Arrays.
>>>>> 
>>>>> The aforementioned motivation will disappear if my ticket
>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 is passed and
>>>>> part of MPI-4.
>>>>> 
>>>>> Another reason is to allow ordered and unordered semantics.  The
>>>>> option to control this happens during window construction and cannot
>>>>> be changed.  If one wants one semantic (unordered) for performance and
>>>>> another (ordered) for correctness, then one needs two windows, one for
>>>>> each.
>>>>> 
>>>>> Jeff
>>>>> 
>>>>> --
>>>>> Jeff Hammond
>>>>> jeff.science at gmail.com
>>>>> http://jeffhammond.github.io/
>>> 
>>> 
>>> 
>>> -- 
>>> Jeff Hammond
>>> jeff.science at gmail.com
>>> http://jeffhammond.github.io/
>