[Mpi-22] Dynamic thread levels

Thu Sep 4 12:24:14 CDT 2008

Dick,

I think there are two independent questions here -- one is about the 
potential performance improvement, and another is about the ease of use 
(or rather complexity of use) for application writers. To some degree, I 
agree with both the concerns, but here are some notes for each one.

With respect to the first, though latency improvement is probably only 
marginal (especially on fast processors), the degradation in the message 
rate is usually quite high.

I've implemented this in MPICH2 and placed a copy here: 
https://svn.mcs.anl.gov/repos/mpi/mpich2/branches/dev/dyn_thread_level. 
Based on this implementation, I did some measurements using a single 
threaded application, but using MPI_Init_thread() vs. MPI_Init().

Experiment 1: Each process sends to MPI_PROC_NULL (emulating an 
infinitely fast network)

MPI_Init -- 29.94 million messages per second
MPI_Init_thread -- 15.86 million messages per second
Difference -- almost 2X

Experiment 2: Two processes communicating over TCP

MPI_Init -- 3.947 million messages per second
MPI_Init_thread -- 3.231 million messages per second
Difference -- about 20%

For fast networks, the difference might be somewhere in between.

With respect to the second question, I agree that we are placing a lot 
of faith in the application developers for this, but this is probably 
not too different from trusting the application in that only one thread 
will call a Barrier or a Bcast, for example. If multiple threads call 
barriers in applications where only one thread is supposed to call, bad 
things can happen, but we trust that the user will not do that. 
Basically, there is a potential for bugs here, but I don't think that's 
any more than what is already present.

If there are any concerns, or if you think we can do this in a better 
way, please do let me know. I'm open to editing this (even substantially 
if needed).

Thanks.

  -- Pavan

On 09/04/2008 09:35 AM, Richard Treumann wrote:
> Pavan,
> 
> Can you explain what improvements you expect if MPI added this?
> 
> If an MPI implementation that is capable of supporting 
> MPI_THREAD_MULTIPLE is running in MPI_THREAD_SINGLE mode, that probably 
> means all protection against multiple threads is being bypassed at 
> rather modest latency savings. The savings probably come from branching 
> over lock operations. Ongoing checking for concurrent threads making MPI 
> calls is expensive enough so doing it would eat up much of the savings 
> from branching over lock operations. Basically, the MPI_THREAD_MULTIPLE 
> capable MPI running in MPI_THREAD_SINGLE mode is probably defenseless 
> against multi thread applications.
> 
> What this means for MPI_Set_thread_level(int required, int * provided);
> 
> is that the application would need to take full responsibility for 
> making sure there is only one thread making MPI calls at the moment a 
> mode switch is requested. If the application gets it wrong there is 
> probably nothing (affordable) the MPI implementation can do to detect 
> the danger. Applications that turned thread safety on and off but were 
> misusing the call would never get an error message and might run without 
> problems 99 times out of 100. 1% of the runs would have mysterious 
> failures and the failures might each look quite different.
> 
> It is certainly possible for an MPI implementation that only supports 
> MPI_THREAD_SINGLE to do some things faster by using simpler global data 
> structures but such an MPI implementation could not honor a runtime 
> request to convert to MPI_THREAD_MULTIPLE. It may even be possible for 
> an MPI implementation at INIT time to chose simple vs. thread safe data 
> structures but again, a switch once everything is up and running is 
> probably not practical.
> 
> This does not seem to me to offer enough payoff to justify the dangers.
> 
> Dick
> 
> Dick Treumann - MPI Team
> IBM Systems & Technology Group
> Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601
> Tele (845) 433-7846 Fax (845) 433-8363
> 
> 
> mpi-22-bounces_at_[hidden] wrote on 09/04/2008 04:30:23 AM:
> 
>  > [image removed]
>  >
>  > [Mpi-22] Dynamic thread levels
>  >
>  > Pavan Balaji
>  >
>  > to:
>  >
>  > mpi-22
>  >
>  > 09/04/2008 04:39 AM
>  >
>  > Sent by:
>  >
>  > mpi-22-bounces_at_[hidden]
>  >
>  > Please respond to "MPI 2.2"
>  >
>  > Hi all,
>  >
>  > I would like to propose an addition of a function call to dynamically
>  > modify the thread level required by the application, instead of at
>  > MPI_Init_thread() time. I'm sending this over email for comments
>  > (attached with this email); I'll upload it on the wiki this evening.
>  >
>  >   -- Pavan
>  >
>  > --
>  > Pavan Balaji
>  > http://www.mcs.anl.gov/~balaji
>  > Dynamic Thread Levels
>  >
>  > Author: Pavan Balaji, Argonne National Laboratory
>  >
>  > Background:
>  >
>  > MPI 2.1 standard allows users to set thread level only at
>  > MPI_Init_thread time. This means that for applications that
>  > dynamically create threads (e.g., hybrid MPI + OpenMP applications),
>  > even non-threaded portions of the application have to rely on the
>  > maximum thread level used in the application (e.g., MULTIPLE).
>  >
>  >
>  > Proposal:
>  >
>  > Add an additional function call to dynamically set thread-level during
>  > the application.
>  >
>  >   int MPI_Set_thread_level(int required, int * provided);
>  >
>  >
>  > Rational:
>  >
>  > The requirement to specify the thread-level at MPI_Init_thread time is
>  > too restrictive for applications that perform a small amount of
>  > communication requiring a high-level of thread support. For
>  > correctness, the standard requires all of the code to follow the same
>  > thread-level, and provides the applications with no way to give the
>  > MPI library more information about their behavior.
>  >
>  >
>  > Impact on MPI implementations:
>  >
>  > Most MPI implementations already provide runtime support for
>  > thread-level, i.e., locks are compiled in, but whether they are
>  > invoked or not is decided at runtime. For implementations that choose
>  > not to respect this option, MPI_Set_thread_level() can just set the
>  > provided level to the current level, by ignoring the required level
>  > specified by the user.
>  >
>  >
>  > Impact on MPI applications:
>  >
>  > Existing MPI applications do not need to be modified at all. But newer
>  > applications can benefit from this additional functionality.
>  > _______________________________________________
>  > mpi-22 mailing list
>  > mpi-22_at_[hidden]
>  > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> mpi-22 mailing list
> mpi-22_at_[hidden]
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22

-- 
Pavan Balaji
http://www.mcs.anl.gov/~balaji