<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 17, 2015 at 4:32 AM, Marc-Andre Hermanns <span dir="ltr"><<a href="mailto:hermanns@jara.rwth-aachen.de" target="_blank">hermanns@jara.rwth-aachen.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Jeff,<br>
<br>
at the moment we don't handle MPI_THREAD_MULTIPLE at all. But we want<br>
to get there ;-)<br>
<br></blockquote><div><br></div><div>You should vote for endpoints, as this may help you out here, particularly if users start mapping endpoints 1:1 w/ threads.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Here is a short recollection of what we do/need. Sorry for the folks<br>
who know/read this already in other context:<br>
<br>
We use, what we call "parallel replay" to analyze large event traces<br>
in parallel. Each thread has its own stream of events, such as enter<br>
and exit for tracking the calling context as well as send and receive<br>
for communication among ranks.<br>
<br>
We analyze on the same scale as the measurement, thus we have one<br>
thread per thread-local trace. Each thread processes its own<br>
thread-local trace. When encountering a communication event, it<br>
re-enacts this communication using the recorded communication<br>
parameters (rank, tag, comm). A send event leads to an issued send, a<br>
receive event leads to an issued receive.<br>
<br>
It is critical that during the analysis, the message matching is<br>
identical to the original application. However, we do not re-enact any<br>
computational time, that is the temporal distance between sends and<br>
receives is certainly different from the original application. As a<br>
consequence, while two sends may have some significant temporal<br>
distance in the original measurement, they could be issued right after<br>
each other during the analysis.<br>
<br>
Markus Geimer and I believe that creating some sort of a sequence<br>
number during measurement could help matching the right messages<br>
during the analysis, as a process could detect that it got mismatched<br>
messages and communicate with other threads to get the correct one.<br>
<br>
<br>
It is unclear, however, how to achieve this:<br>
<br>
a) Sending an additional message within the MPI wrapper at measurement<br>
time may lead to invalid matchings, as the additional message may be<br>
received by a different thread.<br>
<br>
b) Creating a derived datatype on the fly to add tool-level data to<br>
the original payload may induce a large overhead in practically<br>
_every_ send & receive operation and perturb the measurement.<br>
<br></blockquote><div><br></div><div>You should evaluate this experimentally. I wrote a simple test (<a href="https://github.com/jeffhammond/BigMPI/blob/master/test/perf/typepiggy.c">https://github.com/jeffhammond/BigMPI/blob/master/test/perf/typepiggy.c</a>) and measured 1.5 us per call of overhead to create a datatype. That is not significant except for very small messages.</div><div> </div><div>Jeff</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
The best idea in the room so far is some callback mechanism into the<br>
MPI implementation that generates matching information both on sender<br>
and receiver side to generate some form of a sequence number that can<br>
then be saved during measurement. If available on both sender and<br>
receiver this information could then be used to fix incorrect matching<br>
during the analysis.<br>
<br>
Cheers,<br>
Marc-Andre<br>
<span class=""><br>
On 16.12.15 16:40, Jeff Hammond wrote:<br>
> How do you handle MPI_THREAD_MULTIPLE? Understanding what your tool<br>
> does there is a good starting point for this discussion.<br>
><br>
> Jeff<br>
><br>
> On Wed, Dec 16, 2015 at 1:37 AM, Marc-Andre Hermanns<br>
</span>> <<a href="mailto:hermanns@jara.rwth-aachen.de">hermanns@jara.rwth-aachen.de</a> <mailto:<a href="mailto:hermanns@jara.rwth-aachen.de">hermanns@jara.rwth-aachen.de</a>>><br>
<div><div class="h5">> wrote:<br>
><br>
> Hi all,<br>
><br>
> CC: Tools-WG, Markus Geimer (not on either list)<br>
><br>
> sorry for starting a new thread and being so verbose, but I subscribed<br>
> just now. I quoted Dan, Jeff, and Jim from the archive as appropriate.<br>
><br>
> First, let me state that we do not want to prevent this assertion in<br>
> any way. For us as tools provider it is just quite a brain tickler on<br>
> how to support this in our tool and in general.<br>
><br>
> Dan wrote:<br>
> >>> [...] The basic problem is that message matching would be<br>
> >>> non-deterministic and it would be impossible for a tool to show<br>
> >>> the user which receive operation satisfied which send operation<br>
> >>> without internally using some sort of sequence number for each<br>
> >>> send/receive operation. [...]<br>
> >>><br>
> >>> My responses were:<br>
> >>> 1) the user asked for this behaviour so the tool could simply<br>
> >>> gracefully give up the linking function and just state the<br>
> >>> information it knows<br>
> ><br>
> Giving up can only be a temporary solution for tools. The user wants<br>
> to use this advanced feature, thus just saying: "Hey, what you're<br>
> doing is too sophisticated for us. You are on your own now." is not a<br>
> viable long-term strategy.<br>
><br>
> >>> 2) the tool could hook all send and receive operations and<br>
> >>> piggy-back a sequence number into the message header<br>
><br>
> We discussed piggy-backing within the tools group some time in the<br>
> past, but never came to a satisfying way of how to implement this. If,<br>
> in the process of reviving the discussion on a piggy-backing<br>
> interface, we come to a viable solution, it would certainly help with<br>
> the our issues with message matching in general.<br>
><br>
> Scalasca's problem here is that we need to detect (and partly<br>
> recreate) the exact order of message matching to have the correct<br>
> message reach the right receivers.<br>
><br>
> >>> 3) the tool could hook all send and receive operations and<br>
> >>> serialise them to prevent overtaking<br>
><br>
> This is not an option for me. A "performance tool" should strive to<br>
> measure as close to the original behavior as possible. Changing<br>
> communication semantics just to make a tool "work" would have too<br>
> great of an impact on application behavior. After all, if it would<br>
> have only little impact, why should the user choose this option in the<br>
> first place.<br>
><br>
> Jeff wrote:<br>
> >> Remember that one of the use cases of allow_overtaking is<br>
> applications that<br>
> >> have exact matching, in which case allow_overtaking is a way of<br>
> turning off<br>
> >> a feature that isn't used, in order to get a high-performing<br>
> message queue<br>
> >> implementation. In the exact matching case, tools will have no<br>
> problem<br>
> >> matching up sends and recvs.<br>
><br>
> This is true. If the tools can identify this scenario, it could be<br>
> supported by current tools without significant change. However, as it<br>
> is not generally forbidden to have inexact matching (right?), it is<br>
> unclear on how the tools would detect this.<br>
><br>
> What about an additional info key a user can set in this respect:<br>
><br>
> exact_matching => true/false<br>
><br>
> in which the user can state whether it is indeed a scenario of exact<br>
> matching or not. The tool could check this, and issue a warning.<br>
><br>
> >> If tools cannot handle MPI_THREAD_MULTIPLE already, then I<br>
> don't really<br>
> >> care if they can't support this assertion either.<br>
><br>
> Not handling MPI_THREAD_MULTIPLE generally is not carved in stone. ;-)<br>
><br>
> As I said, we (Markus and I) see this as a trigger to come to a viable<br>
> solution for tools like ours to support either situation.<br>
><br>
> >> And in any case, such tools can just intercept the info<br>
> operations and<br>
> >> strip this key if they can't support it.<br>
><br>
> As I wrote above in reply to Dan, stripping options that influence<br>
> behavior is not a good option. I, personally, would rather bail out<br>
> than (silently) change messaging semantics. I can't say what Markus'<br>
> take on this is.<br>
><br>
> Jim wrote:<br>
> > I don't really see any necessary fix to the proposal. We could<br>
> add an<br>
> > advice to users to remind them that they should ensure tools are<br>
> compatible<br>
> > with the info keys. And the reverse advice to tools writers that<br>
> they<br>
> > should check info keys for compatibility.<br>
><br>
> I would second this idea, while emphasizing the burden to be on the<br>
> tool to check for this info key (and potentially others) and warn the<br>
> user of "undersupport".<br>
><br>
> Cheers,<br>
> Marc-Andre<br>
> --<br>
> Marc-Andre Hermanns<br>
> Jülich Aachen Research Alliance,<br>
> High Performance Computing (JARA-HPC)<br>
> Jülich Supercomputing Centre (JSC)<br>
><br>
> Schinkelstrasse 2<br>
> 52062 Aachen<br>
> Germany<br>
><br>
> Phone: +49 2461 61 2509 | +49 241 80 24381<br>
> Fax: +49 2461 80 6 99753<br>
</div></div>> <a href="http://www.jara.org/jara-hpc" rel="noreferrer" target="_blank">www.jara.org/jara-hpc</a> <<a href="http://www.jara.org/jara-hpc" rel="noreferrer" target="_blank">http://www.jara.org/jara-hpc</a>><br>
> email: <a href="mailto:hermanns@jara.rwth-aachen.de">hermanns@jara.rwth-aachen.de</a><br>
> <mailto:<a href="mailto:hermanns@jara.rwth-aachen.de">hermanns@jara.rwth-aachen.de</a>><br>
><br>
><br>
> _______________________________________________<br>
> mpiwg-p2p mailing list<br>
> <a href="mailto:mpiwg-p2p@lists.mpi-forum.org">mpiwg-p2p@lists.mpi-forum.org</a> <mailto:<a href="mailto:mpiwg-p2p@lists.mpi-forum.org">mpiwg-p2p@lists.mpi-forum.org</a>><br>
<span class="">> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p" rel="noreferrer" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p</a><br>
><br>
><br>
><br>
><br>
> --<br>
> Jeff Hammond<br>
</span>> <a href="mailto:jeff.science@gmail.com">jeff.science@gmail.com</a> <mailto:<a href="mailto:jeff.science@gmail.com">jeff.science@gmail.com</a>><br>
> <a href="http://jeffhammond.github.io/" rel="noreferrer" target="_blank">http://jeffhammond.github.io/</a><br>
<div class=""><div class="h5">><br>
><br>
> _______________________________________________<br>
> mpiwg-p2p mailing list<br>
> <a href="mailto:mpiwg-p2p@lists.mpi-forum.org">mpiwg-p2p@lists.mpi-forum.org</a><br>
> <a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p" rel="noreferrer" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p</a><br>
><br>
<br>
--<br>
Marc-Andre Hermanns<br>
Jülich Aachen Research Alliance,<br>
High Performance Computing (JARA-HPC)<br>
Jülich Supercomputing Centre (JSC)<br>
<br>
Schinkelstrasse 2<br>
52062 Aachen<br>
Germany<br>
<br>
Phone: +49 2461 61 2509 | +49 241 80 24381<br>
Fax: +49 2461 80 6 99753<br>
<a href="http://www.jara.org/jara-hpc" rel="noreferrer" target="_blank">www.jara.org/jara-hpc</a><br>
email: <a href="mailto:hermanns@jara.rwth-aachen.de">hermanns@jara.rwth-aachen.de</a><br>
<br>
</div></div><br>_______________________________________________<br>
mpiwg-p2p mailing list<br>
<a href="mailto:mpiwg-p2p@lists.mpi-forum.org">mpiwg-p2p@lists.mpi-forum.org</a><br>
<a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p" rel="noreferrer" target="_blank">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div></div>