<div dir="ltr">Hi Thomas,<div><br></div><div>This sounds like a reasonable proposal to me; it should be possible to create a sane definition for windows on intercommunicators. The RMA interface has focused on intracommunicators, so it could require several additions to e.g. make it possible to query both local and remote groups from the window. I think the main argument against adding this would be the increased complexity in the standard and to implementations. Given that there is a workaround (flattening the communicator to make the window, and using post/start/complete/wait to restrict the set of communication peers in active target), we would need to motivate such a change.</div>
<div><br></div><div>I captured your proposal in a ticket, <a href="https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/412">https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/412</a>, so we remember to discuss this further in the RMA working group.</div>
<div><br></div><div>Best,</div><div> ~Jim.<br>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 22, 2014 at 5:48 AM, Thomas Jahns <span dir="ltr"><<a href="mailto:jahns@dkrz.de" target="_blank">jahns@dkrz.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Hello Jeff,<br>
<div><br>
On 01/21/14 20:03, Jeff Hammond wrote:<br>
> It makes my brain hurt just to think about this.<br>
<br>
</div>I certainly intended no harm, however mild ;-).<br>
<div><br>
> First, one should recognize that RMA communication occurs using the<br>
> window object, not the communicator as in p2p and collectives. The<br>
> reason for a comm object in WIN_CREATE et al. is to enable MPI to<br>
> communicate between participating processes in order to create the<br>
> window. For example, many implementations will do an ALLGATHER inside<br>
> of WIN_CREATE. That operation needs a comm object.<br>
<br>
</div>So there is no problem here with using an intercommunicator?<br>
<div><br>
> The second purpose of the comm is to generate the group of the window<br>
> in order to denote which ranks know about the window. It bothers me a<br>
> negligible amount that the exact definition of the rank arguments for<br>
> RMA is not defined explicitly, but one can reasonably assume that<br>
> those are ranks on the group of the window. I suppose one could talk<br>
> about local and remote groups for a window but again, brain hurt.<br>
<br>
</div>I think what's defined in lines 20-25 of page 420 of MPI 3.0 already makes<br>
perfect sense if comm were an intercommunicator since it's specified in terms of<br>
what would happen for point-to-point communications. So no problem either.<br>
<div><br>
> I do not believe that the communicator has any purpose beyond this and<br>
> thus do not know what the consequences of allowing intercomms would<br>
> be. Perhaps we can use - only for the sake of argument since it is<br>
> not required to have these semantics - that WIN_CREATE is like<br>
> ALLGATHER and use the semantics of ALLGATHER on intercomms to define<br>
> the semantics of win objects created using intercomms.<br>
<br>
</div>For e.g. MPI_Win_post one needs a group, based on the communicator originally<br>
passed to MPI_Win_create but that's already solved for intracommunicators and no<br>
new problem.<br>
<div><br>
> In any case, I don't see any real value in this, honestly. Efficiency<br>
> arguments assume implementation details not specified by the standard.<br>
> Making a promise to not communicate with some ranks has no effect on<br>
> some implementations. Of course we know ones where it does, but<br>
> that's not germane.<br>
<br>
</div>Still it's something I cannot express currently.<br>
<div><br>
> Honestly, I think any sane implementer will just MPI_COMM_TEST_INTER<br>
> -> INTERCOMM_MERGE inside of of WIN_CREATE (et al.), do everything as<br>
> before and then free the temp intracomm at the end. Then I would<br>
> never, ever think about this issue again until someone paid me a huge<br>
> sum of money to optimize for the intercomm case. Thus, the result of<br>
> changing the standard will be almost nothing other than people will be<br>
> able to avoid doing the following explicitly:<br>
<br>
</div>For the above scenario the original communicator needs to be retained, i.e. to<br>
make use of for a later MPI_Group_translate, but otherwise you are right unless<br>
mpi_win_create were to be significantly less expensive with less potential paths.<br>
<div><br>
> int MPE_Win_create(void *base, MPI_Aint size, int disp_unit, MPI_Info<br>
> info, MPI_Comm comm, MPI_Win *win)<br>
> {<br>
> int is_intercomm;<br>
> MPI_Comm intracomm;<br>
> MPI_Comm_test_inter(comm, &is_intercomm);<br>
> if (is_intercomm)<br>
> MPI_Intercomm_merge(comm, 0, &intracomm);<br>
> else<br>
> intracomm = comm;<br>
> MPI_Win_create(base, size, disp_unit, info, intracomm, win);<br>
> if (is_intercomm)<br>
> MPI_Comm_free(&intracomm);<br>
> return MPI_SUCCESS;<br>
> }<br>
><br>
> In short, I don't see any value in trying to define the perverse<br>
> semantics of intercomm-based windows to avoid the ~12 lines of code<br>
> above.<br>
<br>
</div>In my case, where I have a client-server setup with point-to-point communication<br>
happening via an intercommunicator, adding bulk communication via RMA the<br>
current state of affairs means I have to use an additional communicator and<br>
translate IDs. But sure, it can be done with something similar to the above,<br>
it's just more code on the user side.<br>
<div><br>
> In the event we try to act on this proposal, I suppose we use the very<br>
> limited text regarding send-recv on intercomms as our guide: "A target<br>
> process is addressed by its rank in the remote group, both for sends<br>
> and for receives." [pg. 258]<br>
<br>
</div>That's what seemed intuitive to me too (see above).<br>
<br>
Regards, Thomas<br>
<div><div>--<br>
Thomas Jahns<br>
DKRZ GmbH, Department: Application software<br>
<br>
Deutsches Klimarechenzentrum<br>
Bundesstraße 45a<br>
D-20146 Hamburg<br>
<br>
Phone: <a href="tel:%2B49-40-460094-151" value="+4940460094151" target="_blank">+49-40-460094-151</a><br>
Fax: <a href="tel:%2B49-40-460094-270" value="+4940460094270" target="_blank">+49-40-460094-270</a><br>
Email: Thomas Jahns <<a href="mailto:jahns@dkrz.de" target="_blank">jahns@dkrz.de</a>><br>
<br>
</div></div><br>_______________________________________________<br>
mpiwg-rma mailing list<br>
<a href="mailto:mpiwg-rma@lists.mpi-forum.org" target="_blank">mpiwg-rma@lists.mpi-forum.org</a><br>
<a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma</a><br></blockquote></div><br></div></div></div>