[Mpi-comments] Comments on MPI 3.0 draft 2

[Jeremiah Willcock]

These are in reference to the draft at 
<URL:http://meetings.mpi-forum.org/mpi3.0_draft_2.pdf>.  Thank you for 
considering them.

-- Jeremiah Willcock

General:

The line numbers do not match the lines of text; my comments are best 
estimates of which line each line of text is on.

[post 3.0] It might be worthwhile to create a document of "performance 
expectations," basing it on papers such as 
<URL:http://www.mcs.anl.gov/~thakur/papers/qualityofmpi.pdf>, 
<URL:http://www.unixer.de/publications/img/datatype-performance.pdf>, and 
<URL:http://www.mcs.anl.gov/uploads/cels/papers/2008/P1500.pdf>.  This may 
or may not be normative, but maybe you can have a defined 
"self-consistently performing" status for MPI implementations that meet 
the standard and satisfy these extra requirements.

[cosmetic] In many code samples, non-matching curly quotes are used and 
look unusual.  A solution for that is at 
<URL:http://stackoverflow.com/questions/1662037/how-to-write-programming-code-containing-the-character-in-latex>; 
it may be easy to apply globally without needing to change each individual 
quotation mark.

Front matter:

The change log starting on page iii does not have a title.  How much of 
that material is redundant with sections 1.2-1.6?

On page iii line 18, there should be a comma after "Operations".

On page iv lines 6-8, that text seems very obsolete (not correct for the 
current standard).

On page xix line 11, part of the table title is black, while the rest are 
blue.

On page xxviii line 20, "award" should be "awards" and the numbers do not 
usually have number signs before them.  The same issue appears on page xxx 
lines 36-37.

On page xxviii line 29, "frontmatter" should be "front matter".

Chapter 1:

On page 1 line 24, the comma should be after "which" instead of before. 
Also, "methods" being in that list may no longer be relevant.

On page 1 line 44, "coprocessor" should be "coprocessors".

On page 3 line 16, there should be a comma after "e.g.".

On page 3 line 46, the comma should be before the closing quotation mark.

On page 5 line 35, "Info" should be lowercase.

Chapter 2:

On page 9 line 16, why does the MPI 3.0 document still compare itself to 
MPI 1?

On page 9 line 18, "vs" needs a period.

On page 10 line 34, there should be a comma after "OUT".

On page 11 line 30, there should be a comma after "i.e.".

On page 12 line 9, there should not be a comma after "portable".

On page 13 line 43, there should not be a comma after "C".

On page 15, should the MPI version constants be in the first list (of 
compile-time constants)?

On page 18, the font in the table is smaller than the rest of the text or 
other tables in the standard.

On page 19 line 14, the restriction on use of MPI_* names should also 
apply to preprocessor macros, namespaces in C++, types, and other things.

On page 19 line 44, "executing their own code" might be better as "each 
executing its own code".  Is this text trying to say that MPI is not a 
SIMD model?  That is not clear from those two sentences (that the 
executables may or may not be identical, but the processes always have 
different control flow).

Chapter 3:

The top of page 24 refers to "parameters" to mean what chapter 1 said 
would be referred to as "arguments" to avoid confusion with Fortran.

On page 25 lines 36-38, there need to be commas after "MPI_REAL4", 
"MPI_INTEGER1", and "MPI_INTEGER2".

On page 27 line 43, there should not be a space after the ellipsis.

On page 29 lines 35-36, the list of valid values omits "MPI_PROC_NULL" (so 
does the corresponding list for sends).

On page 31 line 32, should "receive" be "send"?

On page 31 lines 38-39, it might be good to have a similar thread safety 
warning about probe and iprobe that is on line 33 about other message 
passing libraries.

On page 32 line 9, there should be a comma after "comm".

On page 33 line 22, the last comma should be a colon.

On page 33 line 48, the example header should be on the same page as the 
code.

On page 34 line 32, "bound" should be "bounds".

On page 36 line 3, "logical or" should be "logical, and".

Is the buffer required to be a valid pointer (not NULL, for example) when 
using count=0 for sends and/or receives?

On page 40 line 39, did you want the "should" to be "must", at least under 
MPI_THREAD_MULTIPLE?  Or is that too strong of a constraint on user-mode 
threading implementations?

On page 44 line 43, "C++" should be removed.

On page 44 line 45, the comma should be before the closing single quote.

On page 46 line 1, doesn't MPI_BSEND_OVERHEAD come into play here as well? 
This paragraph suggests that MPI must be able to use the full user buffer, 
while that is contradicted by page 46 line 40.

Page 47 lines 46-48 seem unclear to me -- is that paragraph an "advice to 
users" that says that an implementation might delay completing standard 
sends until the corresponding receives are posted but it isn't required 
to?  The "may not" wording can also be read to mean that the send is not 
allowed to complete until the receive is finished, which is not what was 
intended.

On page 56 line 30, the text seems to require background progress (sends 
progressing even if MPI is not called).  I know that has been a disputed 
issue in the past, but is that the desired behavior?

On page 57 line 11 and further on, when count = 0 for a multiple 
completion operation, are the arrays passed in guaranteed to not be 
touched by the MPI implementation (and thus they can be NULL)?  I think 
this was implied before, but isn't explicit.

On page 57 line 46, there is an extra space after "request".

On page 60 line 31, it might be good to put "in the same manner" after 
"handled".

On page 62 lines 30-34 (and example 3.16 on page 63), doesn't that text 
contradict the discussion on the mailing list about fairness issues (and 
the lack of fairness guarantees in MPI)?  Also, that text looks like it 
should be under MPI_WAITSOME, not MPI_TESTSOME.

Are the handles passed in to the multiple-completion routines required to 
be distinct?  I.e., can I have the same active handle at two different 
positions in the input array, and if so, what happens?

Does MPI_*PROBE have the same non-overtaking rules as receives (in terms 
of matching the first message sent of those that could match the probe 
operation)?  Page 67 lines 27-38 says that (although possibly not 
normatively since it's in "advice to implementors"), but the 
non-overtaking section doesn't.  Does that rule apply to MPI_IMPROBE as 
well (if I do two MPI_IMPROBES that each match two messages from the same 
sender, the order the probes get the messages must be the same as the send 
order even if I don't do MPI_MRECV before doing the next improbe)?  How do 
those rules interact with MPI_CANCEL on MPI_IMRECV-created requests?

Is there a reason that there are not non-blocking versions of MPI_SENDRECV 
and MPI_SENDRECV_REPLACE?  The second one, in particular, is not trivial 
to emulate using only a single isend/irecv pair.

Chapter 4:

On page 83 line 23, "communication" should be "communications".

On page 83 line 34, "shape and size" should be "shapes and sizes".

On page 84 line 42, "sizeof" should be inside \mathit.

On page 84 line 45, where does "i" come from?  Is the extent rounded to a 
multiple of k_i, or is the upper bound rounded (these differ when lb is 
not aligned)?  If extent is rounded, the +eps term should be moved to the 
computation of extent, not ub, unless that definition of ub is used 
somewhere else.

On page 95, why does MPI_TYPE_CREATE_SUBARRAY use zero-based indexing for 
Fortran; I believe many other places in MPI use one-based array indices 
when called from Fortran, even if they use zero-based indices from C. 
Also, is there a version of MPI_TYPE_CREATE_SUBARRAY that allows strided 
indexing in the dimensions (MPI_TYPE_CREATE_DARRAY looks more limited)?

On page 103 line 18, MPI_ADDRESS has now been removed, not just 
deprecated.

Is offsetof a valid way to get displacements within C structs to use with 
MPI (rather than creating an object of that type and calling 
MPI_GET_ADDRESS on it and its individual fields)?

Wouldn't MPI_TYPE_SIZE and a way to "flatten" an MPI datatype into a 
datatype with the same primitive elements but sequential in memory (or 
using MPI_PACKED) be better than MPI_TYPE_TRUE_EXTENT for the example on 
page 108 line 29?

On page 110 line 24, there should be a comma after "e.g.".

On page 111 line 42, there is some odd space at the end of the line.

On page 112 line 7, there shouldn't be a space after "datatype".

On page 112 lines 11-14, should there be an MPI_TYPE_FREE added as well?

On page 115 line 44, it seems odd that adding MPI_BOTTOM to a valid 
address gives a valid address, since that allows me to create a valid 
address MPI_BOTTOM * n + v where n is nonnegative and v is a valid 
address.

On page 128 line 32, example 4.18 uses "class" as a field name, which 
makes it impossible to compile with a C++ compiler.

Chapter 5:

If there are going to be MPI_Count versions of datatype functions, would 
it at least be possible to have a version of MPI_Alltoallw that uses 
MPI_Aint for displacements?  It seems like that was an oversight before, 
since byte offsets from MPI_BOTTOM are intended as arbitrary addresses.

Chapter 12:

Did the extended generalized request proposal from "Extending the MPI-2 
Generalized Request Interface" by Latham et al not get accepted into MPI 
3.0?

Chapter 16:

On page 597 lines 20-21, "language neutral" should be hyphenated.

On page 597, line 43, "ion" should be "on".

On page 598 line 22, "prototyes" should be "prototypes" in the subsection 
title.

On page 598, tables 16.2 and 16.3 still list C++ bindings.

Chapter 17:

It may be too late to change this, but "typeclass" is an unfortunate name 
to use in MPI since it refers to a specific language feature in Haskell 
and some other functional languages.

On page 600 line 18, "INCLUDE mpif.h" is missing the single quotes that 
are used in other places.

On page 600 line 27, do you want to still mention "C/C++ header files" now 
that the C++ bindings have been removed?

On page 600 line 47, do you want to require Fortran 2008 support for all 
MPI implementations that have any Fortran support at all?  What does that 
mean on platforms that have older Fortran compilers but not Fortran 2008 
support at the compiler level?

On page 601 line 32, should "feature" be "features"?

On page 603 line 2, the period should be before the closing double quote 
(in American English).

On page 604 lines 14-15, should the single quotes be double quotes?

On page 604 line 18, "constaints" should be "constraints".

On page 604 line 36, "With MPI_SUBARRAYS_SUPPORTED==.TRUE." might be 
better as "When MPI_SUBARRAYS_SUPPORTED is defined as .TRUE.".

On page 605 lines 21-23, the italicization of the quotation seems 
incorrect (although I don't know what the source of the quote had 
italicized).

On page 606 lines 16 and 18, "wrong" might read better as "incorrect".

On page 606 line 41, the equation should be in math mode.

On page 607 line 29 and on page 608 line 6, "BIND(C)" should be in the 
code font.

On page 608 line 19, "and" should be removed.

On page 608 lines 24 and 25, "MPI" and "PMPI" use different fonts; this 
may be a typo.

On page 608 line 35, "achieved" should be "available".

On page 608 line 38, "implement" should be "implementation".

On page 609 lines 1-7, the advice to implementors is hard to read (the 
first sentence in particular is very long).

On page 609 line 10, "notifying" should be "indicating".

On page 609 line 33, "MPI_Address" is one of the functions marked as 
removed in MPI 3.0, so it should not be listed there.

On page 610 line 36, should the linker names be in the code font?

On page 611 line 3, "non optional" should be "non-optional".

On page 611 lines 9-10, "compiler" should be "compilers", and the "the" 
before "C" can be removed.

On page 611 lines 16-17, the formatting is strange (extra spaces between 
names).

On page 611 line 44, "b" should be "by".

On page 611 line 48, "MPI_Address" has been removed in MPI 3.0.

On page 612 line 14, the hyphen should be an en dash (denoting a range).

On page 612 lines 37 and 42, "S1", "S2", and "S3" are not in sans serif 
font like they are elsewhere on the page.

On page 613 line 30, the hyphen between the MPI version numbers should be 
an en dash.  Also, why is the MPI 3.0 standard discussing implementing 
older versions?

On page 614 lines 14-15, isn't the second line redundant with the first 
one?

On page 615 line 29, the single quotes should be double quotes.

On page 616 line 14, "IO" should be "I/O".

On page 616 line 45, the subsection title should not have a hyphen before 
"synchronization".

On page 617 lines 13-14, why not just say that the routine must be written 
in an implementation-specific way that achieves the listed behavior, 
rather than specifying a particular requirement about the compiler's 
knowledge of the routine's body?

On page 617 lines 31-32, the single quotes should be double quotes.

On page 618 line 2, there should be a comma after "LOGICAL".

On page 618 line 8, there should be a comma after "COMPLEX".

On page 618 lines 21-22, should the section title referred to be in 
quotes?

On page 619 line 33 and other places, should "external32" be in the code 
font?

On page 621 line 37, there should be commas after "Double" and "8".

On page 622 line 14, there should be a comma after "MPI_UNPACK_EXTERNAL".

On page 624 line 3, "sizeof" is built in to the language as syntax; it is 
not a function.

On page 626 line 13, there is extra space after "90".

On page 626 line 40, the hyphen between the MPI versions should be an en 
dash.

On page 627 line 4, should "KIND" be in the code font?

On page 627 line 30, "mpif.h" may need to be in single quotes.

On page 627 line 39, "module" should be "modules".

On page 628 lines 34-42, does that mean that users are allowed to modify 
send buffers during Fortran nonblocking communications (since the 
implementation is required to act as if it made a copy of the user data 
when the operation began)?  The "Displacements in" sentence on lines 38-39 
is also confusing; does that mean that datatypes need to be created using 
different displacements in Fortran than in C (even if they are 
structurally identical)?  This reading conflicts with page 653, but is 
reasonable from the text on 628.

On page 629 line 2, "comminication" should be "communication".

On page 629 line 21, there should not be a space after the ellipsis.

On page 629 lines 44 and 45, the single quotes should be double quotes.

On page 632 line 27, should the comment be indented to match the one two 
lines later?

On page 632 line 42, "extensions" should be "extension".

On page 633 line 9, there should be a comma before "e.g.".

On page 633 line 12, "as it may fit" sounds odd.

On page 633 lines 47 and 48, the section titles should be in the normal 
text font.

On page 634 line 17, the subsection title has extra space after "and".

On page 634 line 20, "block" should be in the text font.

On page 634 line 41, "be" should be "have been".

On page 635 line 7, there should be spaces after the second two "copy"s.

On page 635 line 9, should "send" be "CALL send"?

On page 635 lines 13 and 16, there should not be spaces around the 
slashes.

On page 635 line 26, there should be a comma after "MPI_RECV".

On page 636 line 36, because of interprocedural optimization, it may not 
be possible for a user routine to have the same behavior as MPI_F_SYNC_REG 
without resorting to compiler-specific techniques, and thus it may not be 
good to recommend doing.  The same thing applies to page 640 lines 15-36 
and page 642 line 48.

On page 637 line 39, should the "ASYNCHRONOUS" in the title be in the code 
font?

On page 639 line 47, the headings should be moved to the same page as the 
code.

On page 640 line 33, "posibilities" should be "possibilities".

On page 643 line 38, "Automatic garbage collection implementation" should 
be "An implementation with automatic garbage collection".

On page 644 line 5, "&" should be in the code font.

On page 647 line 4, remove "applications" from before "library".

On page 648 line 22, is there some reason that the conversion functions 
still take integers on the Fortran side and not the handle types?  Is that 
just a compatibility issue?

On page 648 line 26, "int" should be in the code font.

On page 648 lines 38-39, shouldn't passing in MPI_COMM_NULL in Fortran 
return the C value of MPI_COMM_NULL (and never something else), and the 
behavior be unspecified for passing in an invalid communicator?  The same 
applies to lines 43-44.  Do other predefined communicators need to map to 
the C versions of their predefined values (MPI_COMM_SELF -> MPI_COMM_SELF, 
not something else equivalent to it)?

On page 655 line 3, "stdargs" is not a standard term; use "variable-length 
argument list" instead.

On page 655 line 21, the sentence there does not end with a period (the 
one on "etc." does not count).

On page 655, "address valued" and "integer valued" should have hyphens 
when used as adjectives.

On page 656 line 9, "identical" should be "identically".

On page 657 line 48, the header of the example should be on the same page 
as the code.

On page 658 line 47, "val" should be either sans-serif or in the code 
font.  Some other things in that paragraph might need to be changed as 
well.

On page 659 line 22, "extra state" should be hyphenated in the subsection 
title if it is in the text.

On page 661 line 16, "int" should be in the code font.

Annex A:

On page 674 lines 41-47, don't those constants have particular, 
standard-specified values (not just arbitrary definitions)?

On page 676 lines 10-15, are those values required to be defined as macros 
that expand to integral values?  Thus, they can't have "const int" or 
"unnamed enum" types, but must be macros.

On page 678 line 48 and some places on 679, "module" should be "modules".

On page 685, what are the language bindings for the info keys and values 
supposed to be?

The C prototypes for functions (throughout the standard, but especially 
here) don't end with semicolons like prototypes actually do in C.

On page 693 lines 45-46, shouldn't MPI_COMM_DUP_FN (and the others in 
A.2.4 that are just signatures but not defined by MPI) be moved to section 
A.1.3 with the other user-defined function prototypes?

Annex B:

On page 789 line 28, "errata" is plural, so it should probably be 
"erratum".  There seem to be other places in Annex B with this error as 
well.

On page 789 line 39, the opening quotation mark is backwards.

Section B.1 mentions various C++ bindings to things as having been 
corrected in MPI 3.0, while actually all of those bindings were removed 
(see page 790 lines 5-6 for an example).

On page 790 line 7, the second comma should be inside the quotation marks.

On page 792 line 40, "one-sided" should be lowercase.

On page 793 line 7, there should be a comma after "==".

On page 794 line 17, "a with" should be "with a".

On page 794 line 26, "solution" should be "solutions".

Index:

Page 811 lists now-removed C++ bindings.

The function index starting on page 818 sometimes lists the same function 
twice on the same page (see MPI_ALLTOALLV for an example with page 170).

On page 824, should individual PMPI functions be on the function list? 
Should mpiexec and mpirun?