<div dir="ltr">Rolf,<div><br></div><div>This line is incorrect: MPI_Win_fence(MPI_MODE_NOSTORE + MPI_MODE_NOPRECEDE, win_<span class="">rcv_buf</span>_left );</div><div><p class="">You need to do a bitwise OR of the assertions (MPI_MODE_NOSTORE | MPI_MODE_NOPRECEDE).</p>
<p class="">In halo_1sided_store_win_alloc_shared.c, you are doing stores within the epoch, so MPI_MODE_NOSTORE looks like an incorrect assertion on the closing fence.</p><p class="">Following the Fence epoch, you are reading from the left/right recv buffers. That also needs to be done within an RMA epoch, if you are reading non-local data.</p>
<p class=""> ~Jim.</p></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Feb 21, 2014 at 6:07 AM, Rolf Rabenseifner <span dir="ltr"><<a href="mailto:rabenseifner@hlrs.de" target="_blank">rabenseifner@hlrs.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear member of the RMA group and especially the mpich developers,<br>
<br>
I have real problems with the new shared memory in MPI-3.0,<br>
i.e., the load/stores together with the RMA synchronization<br>
causes wrong execution results.<br>
<br>
The attached<br>
1sided_halo_C_mpich_problems_rabenseifner.tar.gz or .zip<br>
contains<br>
<br>
- 1sided/halo_1sided_put_win_alloc.c<br>
<br>
The basis that works. It uses MPI_Put and MPI_Win_fence for<br>
duplex left/right halo communication.<br>
<br>
- 1sided/halo_1sided_store_win_alloc_shared.c<br>
<br>
This is the same, but a shared memory window is used and<br>
the MPU_Put is substituted by storing the data in the<br>
neighbors window. Same MPI_Win_fence with same assertions.<br>
<br>
This does not work, although I'm sure that my assertions are correct.<br>
<br>
Known possibilities:<br>
- I'm wrong and was not able to understand the assertions<br>
on MPI-3.0 p452:8-19.<br>
- I'm wrong because it is invalid to use the MPI_Win_fence<br>
together with the shared memory windows.<br>
- mpich has a bug.<br>
(The first two possibilities are the reason, why I use this<br>
Forum email list)<br>
<br>
- 1sided/halo_1sided_store_win_alloc_shared_w-a-cray.c<br>
<br>
This is a work-around-for Cray that works on our Cray<br>
and does not use MPI_MODE_NOPRECEDE and MPI_MODE_NOSUCCEED.<br>
It also runs on another mpich installation.<br>
<br>
- 1sided/halo_1sided_store_win_alloc_shared_pscw.c<br>
<br>
Here, MPI_Win_fence is substituted by Post-Start-Complete-Wait<br>
and it does not work for any assertions.<br>
<br>
Same possibilities as above.<br>
<br>
- 1sided/halo_1sided_store_win_alloc_shared_query.c<br>
- 1sided/halo_1sided_store_win_alloc_shared_query_w-a-cray.c<br>
<br>
Same as halo_1sided_store_win_alloc_shared.c<br>
but non-contigues windows are used.<br>
Same problems as above.<br>
<br>
- 1sided/halo_1sided_store_win_alloc_shared_othersync.c<br>
<br>
This version uses the synchronization according to<br>
#413 and it is tested and works on two platforms.<br>
<br>
Best regards<br>
<span class="HOEnZb"><font color="#888888">Rolf<br>
<br>
--<br>
Dr. Rolf Rabenseifner . . . . . . . . . .. email <a href="mailto:rabenseifner@hlrs.de">rabenseifner@hlrs.de</a><br>
High Performance Computing Center (HLRS) . phone <a href="tel:%2B%2B49%280%29711%2F685-65530" value="+4971168565530">++49(0)711/685-65530</a><br>
University of Stuttgart . . . . . . . . .. fax <a href="tel:%2B%2B49%280%29711%20%2F%20685-65832" value="+4971168565832">++49(0)711 / 685-65832</a><br>
Head of Dpmt Parallel Computing . . . <a href="http://www.hlrs.de/people/rabenseifner" target="_blank">www.hlrs.de/people/rabenseifner</a><br>
Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)<br>
</font></span><br>_______________________________________________<br>
mpiwg-rma mailing list<br>
<a href="mailto:mpiwg-rma@lists.mpi-forum.org">mpiwg-rma@lists.mpi-forum.org</a><br>
<a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br></blockquote></div><br></div>