[Mpi3-hybridpm] New draft of the EI chapter uploaded

[Pavan Balaji]

Hi Doug,

On 02/07/2011 07:31 AM, Douglas Miller wrote:
> *balanced *- if set to true, the MPI implementation may assume that all
> threads in the team will actively participate by making MPI calls (or
> calling MPI_HELPER_TEAM_LEAVE). If false, the implementation must allow
> for some threads spending the bulk of their time performing computation
> or other non-MPI activities.

With respect to the first point, threads don't need to "actively 
participate" from the user's perspective. The user just needs to be 
aware that all MPI calls might synchronize all or some of the threads.

For the second point, I think this is incorrect. The threads spending 
the bulk of their time in computation is fine even in "balanced" mode. 
If the user's code is willing to accept the "synchronization overhead" 
during the MPI call, that's fine. For example, we already discussed that 
the following is valid:

MPI_Helper_team_join()
compute(random_time)
pthread_barrier();
if (thread == 0) {
   MPI_SOMETHING();
}
else {
   MPI_SOMETHING_ELSE();
}
MPI_Helper_team_leave()

But the following might be valid too (pthread_barrier is gone):

MPI_Helper_team_join()
compute(random_time)
if (thread == 0) {
   MPI_SOMETHING();
}
else {
   MPI_SOMETHING_ELSE();
}
MPI_Helper_team_leave()

In the second example, if the MPI implementation internally decides to 
synchronize the threads, that's fine, since the application would have 
done that anyway. This should be left to the user. The only thing that 
will be invalid is:

MPI_Helper_team_join()
compute(random_time)
if (thread == 0) {
   MPI_SOMETHING();
}
pthread_barrier();
MPI_Helper_team_leave()

This is invalid because the MPI implementation is expecting to 
synchronize with other threads during the MPI_SOMETHING call, but it 
can't as all other threads are blocked in pthread_barrier().

> ...also, wouldn't setting 'singular' to true also imply/force 'balanced'
> to true? It almost seems like we have a three-state value rather than
> two booleans. From strict to weak: (singular), (!singular,balanced),
> (!singular,!balanced). I'm not clear just how much difference there is
> between singular (true) and balanced (true). Maybe we need only 'balanced'?

Yes, 'singular' is a superset of 'balanced'. I'll point that out. But as 
we discussed in the telecon, 'singular' can give more benefit for 
implementations as they know that other MPI calls will not be scheduled, 
so it's easier to optimize.

  -- Pavan

-- 
Pavan Balaji
http://www.mcs.anl.gov/~balaji