[Mpi3-ft] RTS: Collectives and validate

[Josh Hursey]

It has been proposed that one a process fails in a communicator the posting
of new collectives is disallowed, and the user must create a new, dense
communicator if they want to have access to collective operations again.
This is different than the current proposal where after calling
MPI_Comm_validate the posting of new collective operations is allowed over
the communicator - even though it has 'blanks' in it.

We have strong use cases for being able to call collectives over
communicators with "blanks" in them. So this functionality is required.
However, it was mentioned that a third-party library might be able to 'fake
it' by converting a sparse communicator (exposed to the user) into
operations over the dense communicator (required by MPI). The point would
be to introduce the 'validate/re-enable collectives' semantic as a separate
ticket after the RTS proposal is voted in so as to eliminate the need for
such a library.

There are some problems with the interposition library solution since, for
example, with vector collectives the library would need to do some double
buffering to adjust the sparse data buffer provided by the user into a
dense data buffer that MPI would require. There are other issues, but I
wanted to think through this a bit more before elaborating more on this
point.

One thing I noted on the call was that (from my inspection of the codebase)
FT-MPI uses a dense shadow communicator for many of their collective
operations. In Open MPI, we experimented with a different method in which
we either worked around the failed processes or created a rebalanced
communication tree for collectives over a sparse communicator (we published
the results of this initial study). The point is that by exposing the MPI
library to the sparse communicator (collective over a communicator with
'blanks') the MPI library has more flexibility in how it manages the
collective communication. And as part of that flexibility it does not
require all the badness that the interposition library would have to go
through to provide the same functionality.

So since the additional semantics for collectives are not controversial
(IMHO) and there is a strong use case then why would we not try to get it
right from the beginning?

In short, I have reservations about this proposal, but I wanted more time
to work through the implications. I'll need to post more on that tomorrow
(hopefully). But, as always, comments are welcome in the meantime.

-- Josh

-- 
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://users.nccs.gov/~jjhursey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpi-forum.org/pipermail/mpiwg-ft/attachments/20120125/38ea2d96/attachment.html>