<div dir="ltr"><div>Here is a summary of the meeting and related Slack discussion.</div><div><br></div><div>During the meeting, I presented my proposal for integer and handle constants. However, in response to feedback, I made significant changes to it, which I will describe below.</div><div><br></div><div>There was some debate about using the Huffman code directly versus a table. Joseph wants a table, and one that is smaller than 1024 entries. I have changed the Huffman code so that the maximum table size is not large, and is furthermore amenable to a two-level table that is even smaller, if someone wants to trade time for space.</div><div><br></div><div>We also discussed integer constants, particularly the ones that have interesting positive values, e.g. MPI_BSEND_OVERHEAD. Dan and I agree that we should have a way to query the exact value, which is less than the ABI value, which needs to be an upper bound on all implementations. Dan and I disagree on how to do this. I am going to propose attributes similar to MPI_TAG_UB. Dan or others may propose info or something else. I encourage everyone to look at <a href="https://github.com/mpiwg-abi/specification-text-draft/blob/main/IntegerConstants.md">https://github.com/mpiwg-abi/specification-text-draft/blob/main/IntegerConstants.md</a> and complain about things they don't like.</div><div><br></div><div>After some pondering, I concluded that we should not encode the size of types into handles if the types are not strictly fixed-size by the language. Consider MPI_LONG, for example. Once we fix the platform ABI, e.g. LP64, we know sizeof(long) and thus can encode it in MPI_LONG. However, this would cause two problems. First, the value of MPI_LONG would not be a strict constant and second, third-party languages would need to know the behavior of C to compute the value of MPI_LONG. Lisandro made the point that we create problems for heterogeneous MPI (i.e. ILP32 on one end and LP64 on another) and related activities, and I do not want to break that. Thus, implementations will need to figure out the size of MPI_LONG and related. Fortunately, message rate benchmarks almost always use MPI_CHAR, which is a fixed-size type.</div><div><br></div><div>The latest version of the Huffman code is in <a href="https://github.com/mpiwg-abi/specification-text-draft/pull/6/files">https://github.com/mpiwg-abi/specification-text-draft/pull/6/files</a>, and it has the following features:</div><div>- I made minor changes to the MPI_Op path, because I want to be consistent that MPI_*_NULL is always the zero bit pattern other than the handle type part.</div><div>- Fixed-size datatypes are slightly changed, and exist in the range [576,747].</div><div>- Other datatypes changed a lot, because many C/C++ types are not actually fixed-size. These types are also more table-friendly, as they only span the range [512,572].</div><div><br></div><div>There are a few different implementations that make sense here:</div><div>1. All handles in a single table of 1024 entries (where values is whatever the implementation uses internally: MPICH -> int; OMPI -> pointer), which is ~90% sparse. The worst case storage requirement is 8KiB, which is not onerous.</div><div>2. Datatypes fit into a table of 236 entries, which can be compressed by using an 8-bit hash (internal = table[indirection[ handle & 0b11111111]]) into a table of ~71 entries.</div><div>3. Datatypes use a mostly dense table of 60 entries for non-fixed-size types and something else for the fixed-size types.</div><div>4. The Huffman code is used directly, which requires a handful of register bit mask/shift ops to lookup internal handle values.</div><div><br></div><div>There is an open question about whether we should make the buffer address constants equivalent to zero when they have the same effect as C NULL, or whether we should give them distinct values for debugging purposes.</div><div><br></div><div>I am inclined to make all of the following zero except IN_PLACE, UNWEIGHTED and WEIGHTS_EMPTY, but I am happy to hear feedback to the contrary.</div><div><br></div><div>MPI_STATUS_IGNORE = 0</div><div>MPI_STATUSES_IGNORE = 0</div><div>MPI_ERRCODES_IGNORE = 0</div><div>MPI_ARGV_NULL = 0</div><div>MPI_ARGVS_NULL = 0</div><div>MPI_IN_PLACE != MPI_BOTTOM = 0<br></div><div>MPI_WEIGHTS_EMPTY != MPI_UNWEIGHTED != 0 </div><div><br></div><div>Jeff</div><div><br>On Tue, Apr 4, 2023 at 6:05 PM Jeff Hammond <<a href="mailto:jehammond@nvidia.com">jehammond@nvidia.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg3734655943240883176">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_3734655943240883176WordSection1">
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">I apologize for my incompetence with meeting scheduling and attendance this week. Hopefully this time I am able to get the timing correct
and also attend.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">The topics / decisions of interest are:<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">A. Integer constants<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">We have constants that must be powers of two (mode constants), constants that must have a < relationship (error codes and thread levels),
constants that should be negative to avoid conflicting with e.g. ranks, and constants that can be anything.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">I have a proposal for constants here: <a href="https://github.com/mpiwg-abi/specification-text-draft/blob/main/IntegerConstants.md" target="_blank"><span style="color:rgb(17,85,204)">https://github.com/mpiwg-abi/specification-text-draft/blob/main/IntegerConstants.md</span></a>.
I have not yet implemented Dan's suggestion to make constants unique (presumably, for the aforementioned categories that allow it).<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">I would like feedback on the following:<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">1. Are the values of MPI_MAX_*** acceptable? In all cases, I chose the greater of MPICH and OMPI. For MPI_BSEND_OVERHEAD, I used 512
to be safe. Is that too large? Is 128 better?<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">2. Are there any opponents to Dan's suggestion that all the constants are unique, within reason? If the consensus favors this, I'll redo
all the constants accordingly.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">3. Are there any other integer constant values that bother people? I have very little attachment to any of them, and it is trivial to
change them now.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">B. Handle constants<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">I implemented a Huffman code for these (<a href="https://github.com/mpiwg-abi/specification-text-draft/blob/main/HandleConstants.md" target="_blank"><span style="color:rgb(17,85,204)">https://github.com/mpiwg-abi/specification-text-draft/blob/main/HandleConstants.md</span></a>).
I made a Python <a href="https://github.com/mpiwg-abi/specification-text-draft/blob/main/print-handle-constants.py" target="_blank"><span style="color:rgb(17,85,204)">code</span></a> that implements it, and the program dumps all the values, and can be modified easily
to generate mpi_abi.h.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Does anyone oppose the idea of a Huffman code? I know Hui is indifferent, which is obviously fine. One can ignore the Huffman code and
just view the results as some random values I chose :-)<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">If you like Huffman codes, but dislike mine, then please complain soon. There are some parts that I do not love. For example, fixed size
types are handled consistently and encode their size (as the log_2 of bytes) in bits 3:5, while language default types are on a different branch and encode their size in bits 8:10. I can makes those consistent, but it means the code branches aren't sequential
in the bit indices. I think that's fine, but I am new to Huffman codes.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">An alternative to the above is to say that in mpi_abi.h, that MPI_INT, MPI_LONG and MPI_LONG_LONG are aliased to MPI_INTn_T according to
the ABI definition, and do not exist on their own. This has some appeal, but will change the results of MPI_Type_get_name. What do people think about this?<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">The good news is there is lots of free space in the Huffman code for new handle types and new constants. I am not worried about running
out of space. Already, I have reserved space for a bunch of types that are likely to exist in C and C++ in the near future, so those will be trivial to add later.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Finally, as noted on Slack, we have to figure out whether we reserve space for or standardize some types that OMPI defines today in the
MPI_ namespace. This is the lowest priority for me right now, so if we don't address it this week, that is fine.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Jeff</span></p></div></div></div></blockquote><div> </div></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div></div>