[mpiwg-rma] Synchronization on shared memory windows

Tue Feb 4 11:11:39 CST 2014

To get interprocess memory barrier, right?  Because no RMA means no
sync other than the obvious shared memory consistency business.

Jeff

On Tue, Feb 4, 2014 at 11:01 AM, Balaji, Pavan <balaji at anl.gov> wrote:
> Rolf,
>
> You still need MPI_WIN_SYNC in UNIFIED to ensure memory access ordering.
>
>   — Pavan
>
> On Feb 4, 2014, at 10:59 AM, Rolf Rabenseifner <rabenseifner at hlrs.de> wrote:
>
>> Jeff,
>>
>> thank you for the MPI_FREE_MEM hint. Yes, I'll fix it in my examples.
>>
>> About the synchronization problems:
>> If I use shared memory windows and direct remote load and store
>> instead of the RMA functions PUT or GET,
>> is it then correct, when I never use MPI_WIN-synchronization routines?
>>
>> I would expect this, because in the unified RMA modell,
>> the load and store accesses to the neighbors memory are done
>> directly and MPI is not involved.
>> Because of the unified RMA modell (must be for shared memory windows),
>> there should be no need for cache flush routines.
>> Correct?
>>
>> Therefore, my halo_1sided_put_win_alloc_shared_othersync_20.f90
>> should be correct?
>>
>> Best regards
>> Rolf
>>
>> ----- Original Message -----
>>> From: "Jeff Hammond" <jeff.science at gmail.com>
>>> To: "MPI WG Remote Memory Access working group" <mpiwg-rma at lists.mpi-forum.org>
>>> Cc: "Stefan Andersson" <stefan at cray.com>, "Bill Long" <longb at cray.com>
>>> Sent: Tuesday, February 4, 2014 5:46:28 PM
>>> Subject: Re: [mpiwg-rma] Synchronization on shared memory windows
>>>
>>> Using 2-sided with window memory is no different from load-store.
>>> You
>>> need to synchronize to maintain consistency if you are doing both RMA
>>> and non-RMA on the window.  If you merely use the WIN calls to
>>> allocate memory and only access the private copy, you shouldn't need
>>> any sync.
>>>
>>> I have not explored if your use of active-target sync is valid or not
>>> because I think these modes are almost entirely useless and pay
>>> almost
>>> no attention to their semantics.  This is a matter of opinion that no
>>> one else is required to share, of course.
>>>
>>> Jeff
>>>
>>> On Tue, Feb 4, 2014 at 10:03 AM, Rolf Rabenseifner
>>> <rabenseifner at hlrs.de> wrote:
>>>> Dear all,
>>>>
>>>> I tested MPI-3.0 1-sided synchronization on shared memory windows.
>>>> A lot didn't worked as I expected:
>>>>
>>>> - assertions fail with MPI_Win_fence
>>>>   (there is no restriction defined on MPI-3.0 p452:8-19),
>>>>
>>>> - Post-Start-Complete-Wait fully fails
>>>>   (compare MPI-3.0 p410:16-19, especially the ref to Sect.11.5),
>>>>
>>>> - and MPI_Free_mem also fails for the shared memory windows
>>>>   (compare MI-3.0 p409:23-24, especially that MPI_FREE_MEM
>>>>   is mentioned here).
>>>>
>>>> Attached are some files:
>>>> - halo_1sided_put_win_alloc_20.f90
>>>>
>>>>   This is the basis and works.
>>>>   It is with normal (distributed) windows.
>>>>
>>>> - halo_1sided_put_win_alloc_shared_20.f90
>>>>
>>>>   It is the first shared memory example.
>>>>   It causes several errors with the test on our Cray system:
>>>>    - The assertions on MPI_WIN_FENCE do not work
>>>>    - The MPI_FREE_MEM does not work for the shared buffers
>>>>
>>>>   Is my program wrong? it is a simple left and right
>>>>   1-dim halo exchange.
>>>>
>>>> - halo_1sided_put_win_alloc_shared_20_w-a-cray.f90
>>>>
>>>>   This is a workaround that works on our Cray:
>>>>    - assertions MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED removed
>>>>    - MPI_FREE_MEM removed
>>>>
>>>> - halo_1sided_put_win_alloc_shared_pscw_20.f90
>>>>
>>>>   With Post-Start-Complete-Wait, nothing works!
>>>>   No work-around.
>>>>
>>>> - halo_1sided_put_win_alloc_shared_othersync_20.f90
>>>>
>>>>   In this example, I fully substituted the RMA synchronization
>>>>   by point-to-point synchronization with Irecv, send and waitall.
>>>>   Is this allowed?
>>>>   Was it intended?
>>>>   Is there any wording about in MPI-3.0?
>>>>   Will we have any wording in MPI-next? (3.1 or 4.0)?
>>>>
>>>> I hope, someone knows the answers.
>>>>
>>>> Best regards
>>>> Rolf
>>>>
>>>> --
>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email
>>>> rabenseifner at hlrs.de
>>>> High Performance Computing Center (HLRS) . phone
>>>> ++49(0)711/685-65530
>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 /
>>>> 685-65832
>>>> Head of Dpmt Parallel Computing . . .
>>>> www.hlrs.de/people/rabenseifner
>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room
>>>> 1.307)
>>>>
>>>> _______________________________________________
>>>> mpiwg-rma mailing list
>>>> mpiwg-rma at lists.mpi-forum.org
>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma
>>>
>>>
>>>
>>> --
>>> Jeff Hammond
>>> jeff.science at gmail.com
>>> _______________________________________________
>>> mpiwg-rma mailing list
>>> mpiwg-rma at lists.mpi-forum.org
>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma
>>>
>>
>> --
>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)
>> _______________________________________________
>> mpiwg-rma mailing list
>> mpiwg-rma at lists.mpi-forum.org
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma
>
> _______________________________________________
> mpiwg-rma mailing list
> mpiwg-rma at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

-- 
Jeff Hammond
jeff.science at gmail.com