[mpiwg-p2p] Message matching for tools

Jeff Hammond jeff.science at gmail.com
Thu Dec 17 17:16:05 CST 2015


On Thu, Dec 17, 2015 at 4:32 AM, Marc-Andre Hermanns <
hermanns at jara.rwth-aachen.de> wrote:

> Hi Jeff,
>
> at the moment we don't handle MPI_THREAD_MULTIPLE at all. But we want
> to get there ;-)
>
>
You should vote for endpoints, as this may help you out here, particularly
if users start mapping endpoints 1:1 w/ threads.


> Here is a short recollection of what we do/need. Sorry for the folks
> who know/read this already in other context:
>
> We use, what we call "parallel replay" to analyze large event traces
> in parallel. Each thread has its own stream of events, such as enter
> and exit for tracking the calling context as well as send and receive
> for communication among ranks.
>
> We analyze on the same scale as the measurement, thus we have one
> thread per thread-local trace. Each thread processes its own
> thread-local trace. When encountering a communication event, it
> re-enacts this communication using the recorded communication
> parameters (rank, tag, comm). A send event leads to an issued send, a
> receive event leads to an issued receive.
>
> It is critical that during the analysis, the message matching is
> identical to the original application. However, we do not re-enact any
> computational time, that is the temporal distance between sends and
> receives is certainly different from the original application. As a
> consequence, while two sends may have some significant temporal
> distance in the original measurement, they could be issued right after
> each other during the analysis.
>
> Markus Geimer and I believe that creating some sort of a sequence
> number during measurement could help matching the right messages
> during the analysis, as a process could detect that it got mismatched
> messages and communicate with other threads to get the correct one.
>
>
> It is unclear, however, how to achieve this:
>
> a) Sending an additional message within the MPI wrapper at measurement
> time may lead to invalid matchings, as the additional message may be
> received by a different thread.
>
> b) Creating a derived datatype on the fly to add tool-level data to
> the original payload may induce a large overhead in practically
> _every_ send & receive operation and perturb the measurement.
>
>
You should evaluate this experimentally.  I wrote a simple test (
https://github.com/jeffhammond/BigMPI/blob/master/test/perf/typepiggy.c)
and measured 1.5 us per call of overhead to create a datatype.  That is not
significant except for very small messages.

Jeff

The best idea in the room so far is some callback mechanism into the
> MPI implementation that generates matching information both on sender
> and receiver side to generate some form of a sequence number that can
> then be saved during measurement. If available on both sender and
> receiver this information could then be used to fix incorrect matching
> during the analysis.
>
> Cheers,
> Marc-Andre
>
> On 16.12.15 16:40, Jeff Hammond wrote:
> > How do you handle MPI_THREAD_MULTIPLE?  Understanding what your tool
> > does there is a good starting point for this discussion.
> >
> > Jeff
> >
> > On Wed, Dec 16, 2015 at 1:37 AM, Marc-Andre Hermanns
> > <hermanns at jara.rwth-aachen.de <mailto:hermanns at jara.rwth-aachen.de>>
> > wrote:
> >
> >     Hi all,
> >
> >     CC: Tools-WG, Markus Geimer (not on either list)
> >
> >     sorry for starting a new thread and being so verbose, but I
> subscribed
> >     just now. I quoted Dan, Jeff, and Jim from the archive as
> appropriate.
> >
> >     First, let me state that we do not want to prevent this assertion in
> >     any way. For us as tools provider it is just quite a brain tickler on
> >     how to support this in our tool and in general.
> >
> >     Dan wrote:
> >     >>> [...] The basic problem is that message matching would be
> >     >>> non-deterministic and it would be impossible for a tool to show
> >     >>> the user which receive operation satisfied which send operation
> >     >>> without internally using some sort of sequence number for each
> >     >>> send/receive operation. [...]
> >     >>>
> >     >>> My responses were:
> >     >>> 1) the user asked for this behaviour so the tool could simply
> >     >>> gracefully give up the linking function and just state the
> >     >>> information it knows
> >     >
> >     Giving up can only be a temporary solution for tools. The user wants
> >     to use this advanced feature, thus just saying: "Hey, what you're
> >     doing is too sophisticated for us. You are on your own now." is not a
> >     viable long-term strategy.
> >
> >     >>> 2) the tool could hook all send and receive operations and
> >     >>> piggy-back a sequence number into the message header
> >
> >     We discussed piggy-backing within the tools group some time in the
> >     past, but never came to a satisfying way of how to implement this.
> If,
> >     in the process of reviving the discussion on a piggy-backing
> >     interface, we come to a viable solution, it would certainly help with
> >     the our issues with message matching in general.
> >
> >     Scalasca's problem here is that we need to detect (and partly
> >     recreate) the exact order of message matching to have the correct
> >     message reach the right receivers.
> >
> >     >>> 3) the tool could hook all send and receive operations and
> >     >>> serialise them to prevent overtaking
> >
> >     This is not an option for me. A "performance tool" should strive to
> >     measure as close to the original behavior as possible. Changing
> >     communication semantics just to make a tool "work" would have too
> >     great of an impact on application behavior. After all, if it would
> >     have only little impact, why should the user choose this option in
> the
> >     first place.
> >
> >     Jeff wrote:
> >     >> Remember that one of the use cases of allow_overtaking is
> >     applications that
> >     >> have exact matching, in which case allow_overtaking is a way of
> >     turning off
> >     >> a feature that isn't used, in order to get a high-performing
> >     message queue
> >     >> implementation. In the exact matching case, tools will have no
> >     problem
> >     >> matching up sends and recvs.
> >
> >     This is true. If the tools can identify this scenario, it could be
> >     supported by current tools without significant change. However, as it
> >     is not generally forbidden to have inexact matching (right?), it is
> >     unclear on how the tools would detect this.
> >
> >     What about an additional info key a user can set in this respect:
> >
> >     exact_matching => true/false
> >
> >     in which the user can state whether it is indeed a scenario of exact
> >     matching or not. The tool could check this, and issue a warning.
> >
> >     >> If tools cannot handle MPI_THREAD_MULTIPLE already, then I
> >     don't really
> >     >> care if they can't support this assertion either.
> >
> >     Not handling MPI_THREAD_MULTIPLE generally is not carved in stone.
> ;-)
> >
> >     As I said, we (Markus and I) see this as a trigger to come to a
> viable
> >     solution for tools like ours to support either situation.
> >
> >     >> And in any case, such tools can just intercept the info
> >     operations and
> >     >> strip this key if they can't support it.
> >
> >     As I wrote above in reply to Dan, stripping options that influence
> >     behavior is not a good option. I, personally, would rather bail out
> >     than (silently) change messaging semantics. I can't say what Markus'
> >     take on this is.
> >
> >     Jim wrote:
> >     > I don't really see any necessary fix to the proposal. We could
> >     add an
> >     > advice to users to remind them that they should ensure tools are
> >     compatible
> >     > with the info keys. And the reverse advice to tools writers that
> >     they
> >     > should check info keys for compatibility.
> >
> >     I would second this idea, while emphasizing the burden to be on the
> >     tool to check for this info key (and potentially others) and warn the
> >     user of "undersupport".
> >
> >     Cheers,
> >     Marc-Andre
> >     --
> >     Marc-Andre Hermanns
> >     Jülich Aachen Research Alliance,
> >     High Performance Computing (JARA-HPC)
> >     Jülich Supercomputing Centre (JSC)
> >
> >     Schinkelstrasse 2
> >     52062 Aachen
> >     Germany
> >
> >     Phone: +49 2461 61 2509 | +49 241 80 24381
> >     Fax: +49 2461 80 6 99753
> >     www.jara.org/jara-hpc <http://www.jara.org/jara-hpc>
> >     email: hermanns at jara.rwth-aachen.de
> >     <mailto:hermanns at jara.rwth-aachen.de>
> >
> >
> >     _______________________________________________
> >     mpiwg-p2p mailing list
> >     mpiwg-p2p at lists.mpi-forum.org <mailto:mpiwg-p2p at lists.mpi-forum.org>
> >     http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p
> >
> >
> >
> >
> > --
> > Jeff Hammond
> > jeff.science at gmail.com <mailto:jeff.science at gmail.com>
> > http://jeffhammond.github.io/
> >
> >
> > _______________________________________________
> > mpiwg-p2p mailing list
> > mpiwg-p2p at lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p
> >
>
> --
> Marc-Andre Hermanns
> Jülich Aachen Research Alliance,
> High Performance Computing (JARA-HPC)
> Jülich Supercomputing Centre (JSC)
>
> Schinkelstrasse 2
> 52062 Aachen
> Germany
>
> Phone: +49 2461 61 2509 | +49 241 80 24381
> Fax: +49 2461 80 6 99753
> www.jara.org/jara-hpc
> email: hermanns at jara.rwth-aachen.de
>
>
> _______________________________________________
> mpiwg-p2p mailing list
> mpiwg-p2p at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-p2p
>



-- 
Jeff Hammond
jeff.science at gmail.com
http://jeffhammond.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpi-forum.org/pipermail/mpiwg-p2p/attachments/20151217/a667046e/attachment-0001.html>


More information about the mpiwg-p2p mailing list