[mpi3-coll] Nonblocking collectives standard draft

[Torsten Hoefler]

Hello Christian,
thanks for the review of the first draft. All comments that I deleted
are fixed, but I'd like to open the discussion of others up.

> btw. all line numbers within the MPI standard are not really line 
> numbers (they have equal distances and as soon as there is another font 
> size - like for the interface definitions - then you end up anywhere...)
>   It's like "look at page X at _centimetre_ Y".
yes, I sent a note about this to the MPI-2.1 mailinglist a while ago,
but saw no reaction.

> p 49, l 46: it is not enough to state "receive buffer must not be read"
>   it should be forbidden to access the receive buffer in any way
>   (imagine an implementation that uses the buffer to forwards sth.)
Right, I changed this to be similar to MPI-2.1 p52 line 4-7. Now it
reads "must not be accessed". 

> p 50, l 7: we should discuss MPI_REQUEST_FREE (again) - I talked to some
>   colleagues and we see application scenarios where this might be
>   useful (e.g., if only a subgroup participates in an irregular
>   collective); because this is a local operation, we also don't
>   see any problems by generally allowing MPI_REQUEST_FREE for
>   nonblocking collective operations
I personally don't like request free because it makes thread-safe
implementations really hard. Also, I don't see a good portable way to
check/ensure that the operation completed on the receiver side (we had
this discussion before on the MPI-2.1 ML I think). Do you know one?

> p 50, l 19: maybe we should be even more precise and add after 
> "environments." something like "It is therefore erroneous to start a set 
> of nbc in different logical orders on the participating ranks."
we didn't really define "different logical order" anywhere. This is why
we decided to go by examples.

>   The sentence in l 20 ("The behavior...") can and should be deleted.
>   "Nbc operations and bc operations [+are totally unrelated and ]
>    do not match each other."
addition not made ("totally unrelated") - I think we shouldn't say this
sinde they perform very similar operations and can be implemented on top
of each other.

> several locations: "memory movement" should be replaced by "data movement"
>   (but "movement" sounds bad anyway because you say
>    "after completion" - maybe "data placement" would be
>    more appropriate)
the MPI standard says for example "The ``in place'' operations are
provided to reduce unnecessary memory motion". I agree that it sounds
strange, but it seems to be the used like this in MPI terms. But I am
open to change it.

> p 61, l 31.8 & p 62, l 27.5: "after it completion" -> "after completion"?
I don't see a mistake there. It says "after it completed" in the text.

> btw. there seems to be a potential naming problem for beginners:
>   "Exclusive Scan -> Exscan" && "Inclusive Scan -> Iscan"???
Hmm, I wouldn't think so, but we can discuss this.

> p 69, l 41: "in SPMD style programs" seems superflous to me
>   but "double buffering" could also be added as an example ...
it's not superflous. One can also implement task-parallel or other
programming paradigms with MPI. I added double-bufferng (even though
this is just a weaker form of pipelining).

> Example 5.33 seems to be very dangerous for at least two reasons:
>   2) this doesn't make it a "global" allreduce because
>      (depending how sbuf and rbuf are initialized) all
>      seem to calculate different results
>      (0+0+1+2 vs. 0+1+1+2 vs. 1+2+0+2)
exactly - that's what they want to do. Everybody wants to reduce in two
communicators. This tiny example is probably not very useful, but easy to
understand. Think of nearest neighbor reductions.

>      This example is a good idea but needs some clarifications...
> 
> What do we want to have (either or):
>   a) the same ordering over both blocking and nonblocking collectives
>      (suggested by Example 5.29)
>   b) blocking and nonblocking collectives are completely independent
>      (suggested by "reserved tag-space" p 50, l 15)
> Upon a decision, the related text needs some clarification.
we decided to go with a). The advice to implementors talks only about
non-blocking point-to-point when it mentions the reserved tag space.

> All in all a big step towards the standardization. However, there seem 
> to be quite a couple of mistakes in the copy-and-paste parts and I'm 
> quite sure that I didn't catch all in my first go... so please be 
> especially careful in those places.
yes, this is the first draft to be circulated only in the workgroup and
far away from an official reading. I attached an updated document to the
wiki page [1] and put another convenience copy here [2].

Thank you very much for your review!

All the Best,
  Torsten

[1]: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/NBColl
[2]: http://www.unixer.de/sec/nbc-proposal-11-13-2008.pdf

-- 
 bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ -----
Torsten Hoefler       | Postdoctoral Researcher
Open Systems Lab      | Indiana University    
150 S. Woodlawn Ave.  | Bloomington, IN, 474045, USA
Lindley Hall Room 135 | +01 (812) 855-3608