<div dir="ltr">We should just explicitly deprecate support for segmented addressing so we can stop thinking about it. Nobody is ever going to run MPI in x86 real mode. 99% of HPC is done with LP64, which doesn't support segmentation, and in ILP32 mode, there isn't a large-count problem.<div><br></div><div><span style="font-family:sans-serif;font-size:14px">"The </span>x86-64<span style="font-family:sans-serif;font-size:14px"> architecture, introduced in 2003, has largely dropped support for segmentation in 64-bit mode."</span><br></div><div><span style="font-family:sans-serif;font-size:14px"><br></span></div><div><span style="font-family:sans-serif;font-size:14px">Jeff </span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 24, 2019 at 9:42 AM HOLMES Daniel via mpiwg-large-counts <<a href="mailto:mpiwg-large-counts@lists.mpi-forum.org">mpiwg-large-counts@lists.mpi-forum.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;">
Hi Rolf & Jeff,
<div><br>
</div>
<div>I think this wiki article is instructive on this topic also:</div>
<div><a href="https://en.wikipedia.org/wiki/X86_memory_segmentation" target="_blank">https://en.wikipedia.org/wiki/X86_memory_segmentation</a></div>
<div><br>
</div>
<div>This seems like a crazy memory addressing system to me personally, but it is a (historic) example of a segmented addressing approach that MPI_Aint can support.</div>
<div><br>
</div>
<div>The “strange properties” for arithmetic are strange indeed, depending on what the MPI_Aint stores and how.</div>
<div><br>
</div>
<div>If MPI_Aint was 20 bits long and stores only the address, then it cannot be used to determine uniquely which segment is being used or what the offset is within that segment (there are 4096 possible answers). Does MPI need that more detailed information?
Probably - because segments were a way of implementing memory protection, i.e. accessing a segment you did not have permission to access led to a “segmentation fault” error. I do not know enough about these old architectures to say whether an attempt to access
the *same byte* using two different segment:offset pairs that produce the *same* address could result in different behaviour. That is, if I have access permissions for segment 3 but not for segment 4, I can access {seg=3,offset=2^16-16} but can I access {segment=4,offset=2^16-32},
which is the same byte? If not, then MPI needs to store segment and offset inside MPI_Aint to be able to check and to set registers correctly.</div>
<div><br>
</div>
<div>If MPI_Aint is 32 bits long and stores the segment in the first 16 bits and the offset in the last 16 bits, then the 20 bit address can be computed in a single simple instruction and both segment and offset are immediately retrievable. However,
doing ordinary arithmetic with this bitwise representation is unwise because it is a compound structure type. Let us subtract 1 from an MPI_Aint of this layout which stores offset of 0 and some non-zero segment. We get offset (2^16-1) in segment (s-1), which
is not 1 byte before the previous MPI_Aint because segments overlap. The same happens when adding and overflowing the offset portion - it changes the segment in an incorrect way. Segment++ moves the address forward only 16 bytes, not 2^16 bytes.</div>
<div><br>
</div>
<div>The wrap-around from the end of the address space back to the beginning is also a source of strange properties for arithmetic.</div>
<div><br>
</div>
<div>One of the key statements from that wiki page is this:</div>
<div><br>
</div>
<div>
<blockquote type="cite">The root of the problem is that no appropriate address-arithmetic instructions suitable for flat addressing of the entire memory range are available.[citation needed] Flat addressing is possible by applying multiple instructions,
which however leads to slower programs.<br>
</blockquote>
<div>
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div dir="auto" style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br>
Cheers,</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
Dan.</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
—<br>
Dr Daniel Holmes PhD<br>
Architect (HPC Research)<br>
<a href="mailto:d.holmes@epcc.ed.ac.uk" target="_blank">d.holmes@epcc.ed.ac.uk</a><br>
Phone: +44 (0) 131 651 3465<br>
Mobile: +44 (0) 7940 524 088<br>
Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
—</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
—</div>
</div>
</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On 24 Oct 2019, at 17:27, Jeff Squyres (jsquyres) via mpiwg-large-counts <<a href="mailto:mpiwg-large-counts@lists.mpi-forum.org" target="_blank">mpiwg-large-counts@lists.mpi-forum.org</a>> wrote:</div>
<br>
<div>
<div style="overflow-wrap: break-word;">
On Oct 24, 2019, at 11:15 AM, Rolf Rabenseifner <<a href="mailto:rabenseifner@hlrs.de" target="_blank">rabenseifner@hlrs.de</a>> wrote:<br>
<div>
<blockquote type="cite"><br>
<div><span style="font-family:Menlo-Regular;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">For
me, it looked like that there was some misunderstanding</span><br style="font-family:Menlo-Regular;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span style="font-family:Menlo-Regular;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">of
the concept that absolute and relative addresses</span><br style="font-family:Menlo-Regular;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
<span style="font-family:Menlo-Regular;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline">and
number of bytes that can be stored in MPI_Aint. </span><br style="font-family:Menlo-Regular;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">
</div>
</blockquote>
</div>
<div><br>
</div>
<div>...with the caveat that MPI_Aint -- as it is right now -- does not support modern segmented memory systems (i.e., where you need more than a small number of bits to indicate the segment where the memory lives).</div>
<div><br>
</div>
<div>I think that changes the conversation entirely, right?</div>
<div><span><br>
-- <br>
</span><span>Jeff Squyres<br>
</span><span><a href="mailto:jsquyres@cisco.com" target="_blank">jsquyres@cisco.com</a></span>
</div>
<br>
</div>
_______________________________________________<br>
mpiwg-large-counts mailing list<br>
<a href="mailto:mpiwg-large-counts@lists.mpi-forum.org" target="_blank">mpiwg-large-counts@lists.mpi-forum.org</a><br>
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts" target="_blank">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
_______________________________________________<br>
mpiwg-large-counts mailing list<br>
<a href="mailto:mpiwg-large-counts@lists.mpi-forum.org" target="_blank">mpiwg-large-counts@lists.mpi-forum.org</a><br>
<a href="https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts" rel="noreferrer" target="_blank">https://lists.mpi-forum.org/mailman/listinfo/mpiwg-large-counts</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Jeff Hammond<br><a href="mailto:jeff.science@gmail.com" target="_blank">jeff.science@gmail.com</a><br><a href="http://jeffhammond.github.io/" target="_blank">http://jeffhammond.github.io/</a></div>