<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Jeff - All your changes make reasonable sense to me.
<div><br>
</div>
<div><span class="Apple-tab-span" style="white-space:pre"></span>Quincey</div>
<div><br>
<div><br>
<blockquote type="cite">
<div>On Apr 12, 2023, at 2:30 AM, Jeff Hammond <jeff.science@gmail.com> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div>
<div class="WordSection1">
<table class="MsoTableGrid" border="1" cellspacing="0" cellpadding="0" style="border-collapse:collapse;border:none">
<tbody>
<tr style="height:15.25pt">
<td width="711" valign="top" style="width:842.35pt;border:solid #ED7D31 1.5pt;padding:0in 5.4pt 0in 5.4pt;height:15.25pt">
<p><strong><span style="background:#FFFF99">CAUTION</span></strong><span style="background:#FFFF99">: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</div>
<br>
<div>
<div dir="ltr">
<div>Here is a summary of the meeting and related Slack discussion.</div>
<div><br>
</div>
<div>During the meeting, I presented my proposal for integer and handle constants. However, in response to feedback, I made significant changes to it, which I will describe below.</div>
<div><br>
</div>
<div>There was some debate about using the Huffman code directly versus a table. Joseph wants a table, and one that is smaller than 1024 entries. I have changed the Huffman code so that the maximum table size is not large, and is furthermore amenable to a
two-level table that is even smaller, if someone wants to trade time for space.</div>
<div><br>
</div>
<div>We also discussed integer constants, particularly the ones that have interesting positive values, e.g. MPI_BSEND_OVERHEAD. Dan and I agree that we should have a way to query the exact value, which is less than the ABI value, which needs to be an upper
bound on all implementations. Dan and I disagree on how to do this. I am going to propose attributes similar to MPI_TAG_UB. Dan or others may propose info or something else. I encourage everyone to look at <a href="https://github.com/mpiwg-abi/specification-text-draft/blob/main/IntegerConstants.md">https://github.com/mpiwg-abi/specification-text-draft/blob/main/IntegerConstants.md</a>
and complain about things they don't like.</div>
<div><br>
</div>
<div>After some pondering, I concluded that we should not encode the size of types into handles if the types are not strictly fixed-size by the language. Consider MPI_LONG, for example. Once we fix the platform ABI, e.g. LP64, we know sizeof(long) and thus
can encode it in MPI_LONG. However, this would cause two problems. First, the value of MPI_LONG would not be a strict constant and second, third-party languages would need to know the behavior of C to compute the value of MPI_LONG. Lisandro made the point
that we create problems for heterogeneous MPI (i.e. ILP32 on one end and LP64 on another) and related activities, and I do not want to break that. Thus, implementations will need to figure out the size of MPI_LONG and related. Fortunately, message rate benchmarks
almost always use MPI_CHAR, which is a fixed-size type.</div>
<div><br>
</div>
<div>The latest version of the Huffman code is in <a href="https://github.com/mpiwg-abi/specification-text-draft/pull/6/files">https://github.com/mpiwg-abi/specification-text-draft/pull/6/files</a>, and it has the following features:</div>
<div>- I made minor changes to the MPI_Op path, because I want to be consistent that MPI_*_NULL is always the zero bit pattern other than the handle type part.</div>
<div>- Fixed-size datatypes are slightly changed, and exist in the range [576,747].</div>
<div>- Other datatypes changed a lot, because many C/C++ types are not actually fixed-size. These types are also more table-friendly, as they only span the range [512,572].</div>
<div><br>
</div>
<div>There are a few different implementations that make sense here:</div>
<div>1. All handles in a single table of 1024 entries (where values is whatever the implementation uses internally: MPICH -> int; OMPI -> pointer), which is ~90% sparse. The worst case storage requirement is 8KiB, which is not onerous.</div>
<div>2. Datatypes fit into a table of 236 entries, which can be compressed by using an 8-bit hash (internal = table[indirection[ handle & 0b11111111]]) into a table of ~71 entries.</div>
<div>3. Datatypes use a mostly dense table of 60 entries for non-fixed-size types and something else for the fixed-size types.</div>
<div>4. The Huffman code is used directly, which requires a handful of register bit mask/shift ops to lookup internal handle values.</div>
<div><br>
</div>
<div>There is an open question about whether we should make the buffer address constants equivalent to zero when they have the same effect as C NULL, or whether we should give them distinct values for debugging purposes.</div>
<div><br>
</div>
<div>I am inclined to make all of the following zero except IN_PLACE, UNWEIGHTED and WEIGHTS_EMPTY, but I am happy to hear feedback to the contrary.</div>
<div><br>
</div>
<div>MPI_STATUS_IGNORE = 0</div>
<div>MPI_STATUSES_IGNORE = 0</div>
<div>MPI_ERRCODES_IGNORE = 0</div>
<div>MPI_ARGV_NULL = 0</div>
<div>MPI_ARGVS_NULL = 0</div>
<div>MPI_IN_PLACE != MPI_BOTTOM = 0<br>
</div>
<div>MPI_WEIGHTS_EMPTY != MPI_UNWEIGHTED != 0 </div>
<div><br>
</div>
<div>Jeff</div>
<div><br>
On Tue, Apr 4, 2023 at 6:05 PM Jeff Hammond <<a href="mailto:jehammond@nvidia.com">jehammond@nvidia.com</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="msg3734655943240883176">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="m_3734655943240883176WordSection1">
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">I apologize for my incompetence with meeting scheduling and attendance this week. Hopefully this time I am able to get the timing correct
and also attend.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">The topics / decisions of interest are:<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">A. Integer constants<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">We have constants that must be powers of two (mode constants), constants that must have a < relationship (error codes and thread levels),
constants that should be negative to avoid conflicting with e.g. ranks, and constants that can be anything.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">I have a proposal for constants here: <a href="https://github.com/mpiwg-abi/specification-text-draft/blob/main/IntegerConstants.md" target="_blank"><span style="color:rgb(17,85,204)">https://github.com/mpiwg-abi/specification-text-draft/blob/main/IntegerConstants.md</span></a>.
I have not yet implemented Dan's suggestion to make constants unique (presumably, for the aforementioned categories that allow it).<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">I would like feedback on the following:<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">1. Are the values of MPI_MAX_*** acceptable? In all cases, I chose the greater of MPICH and OMPI. For MPI_BSEND_OVERHEAD, I used 512
to be safe. Is that too large? Is 128 better?<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">2. Are there any opponents to Dan's suggestion that all the constants are unique, within reason? If the consensus favors this, I'll
redo all the constants accordingly.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">3. Are there any other integer constant values that bother people? I have very little attachment to any of them, and it is trivial to
change them now.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">B. Handle constants<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">I implemented a Huffman code for these (<a href="https://github.com/mpiwg-abi/specification-text-draft/blob/main/HandleConstants.md" target="_blank"><span style="color:rgb(17,85,204)">https://github.com/mpiwg-abi/specification-text-draft/blob/main/HandleConstants.md</span></a>).
I made a Python <a href="https://github.com/mpiwg-abi/specification-text-draft/blob/main/print-handle-constants.py" target="_blank"><span style="color:rgb(17,85,204)">code</span></a> that implements it, and the program dumps all the values, and can be modified
easily to generate mpi_abi.h.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Does anyone oppose the idea of a Huffman code? I know Hui is indifferent, which is obviously fine. One can ignore the Huffman code
and just view the results as some random values I chose :-)<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">If you like Huffman codes, but dislike mine, then please complain soon. There are some parts that I do not love. For example, fixed
size types are handled consistently and encode their size (as the log_2 of bytes) in bits 3:5, while language default types are on a different branch and encode their size in bits 8:10. I can makes those consistent, but it means the code branches aren't sequential
in the bit indices. I think that's fine, but I am new to Huffman codes.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">An alternative to the above is to say that in mpi_abi.h, that MPI_INT, MPI_LONG and MPI_LONG_LONG are aliased to MPI_INTn_T according
to the ABI definition, and do not exist on their own. This has some appeal, but will change the results of MPI_Type_get_name. What do people think about this?<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">The good news is there is lots of free space in the Huffman code for new handle types and new constants. I am not worried about running
out of space. Already, I have reserved space for a bunch of types that are likely to exist in C and C++ in the near future, so those will be trivial to add later.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Finally, as noted on Slack, we have to figure out whether we reserve space for or standardize some types that OMPI defines today in the
MPI_ namespace. This is the lowest priority for me right now, so if we don't address it this week, that is fine.<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:12pt;font-family:Arial,sans-serif;color:rgb(34,34,34)">Jeff</span></p>
</div>
</div>
</div>
</blockquote>
<div> </div>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">Jeff Hammond<br>
<a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br>
<a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>
</div>
</div>
</div>
-- <br>
mpiwg-abi mailing list<br>
mpiwg-abi@lists.mpi-forum.org<br>
https://lists.mpi-forum.org/mailman/listinfo/mpiwg-abi<br>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>