<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hi Rolf,
<div class=""><br class="">
</div>
<div class="">Thanks. I’ve copied your response into the the issue to keep it all in one place (so we don’t have to search email to find it).<br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="Apple-interchange-newline">
Cheers,</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Dan.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Architect (HPC Research)<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
—</div>
</div>
</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 5 Nov 2019, at 17:58, Rolf Rabenseifner <<a href="mailto:rabenseifner@hlrs.de" class="">rabenseifner@hlrs.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Sorry, I now detected that you wanted to point me also to<br class="">
<br class="">
<blockquote type="cite" class="">
<blockquote type="cite" class="">----- Jeff Hammond <<a href="mailto:jeff.science@gmail.com" class="">jeff.science@gmail.com</a><<a href="mailto:jeff.science@gmail.com" class="">mailto:jeff.science@gmail.com</a>>><br class="">
wrote:<br class="">
Rolf:<br class="">
<br class="">
Have you looked at<br class="">
<a href="https://github.com/mpiwg-large-count/large-count-issues/issues/6" class="">https://github.com/mpiwg-large-count/large-count-issues/issues/6</a> ?<br class="">
</blockquote>
</blockquote>
<br class="">
where I found your text. Thank you for remembering me.<br class="">
<br class="">
About your choices:<br class="">
<br class="">
<blockquote type="cite" class="">1.change the name of MPI_Type_get_extext_x to MPI_Type_get_extext_l to fit in with all the other _l functions for large count. This is a bad idea because the MPI_Count variant is not (just) embiggening the MPI_Aint variant,
it is providing a different role (target files, not absolute memory).<br class="">
</blockquote>
<br class="">
Agreed, bad idea.<br class="">
<br class="">
<blockquote type="cite" class="">2.add a new MPI_Type_get_extext_l that uses MPI_Offset. This is superfluous because we already have the _x variant with MPI_Count.<br class="">
</blockquote>
<br class="">
Agreed, bad idea, because there is no "int Count" Argument.<br class="">
<br class="">
<blockquote type="cite" class="">3.leave it well alone. This seems like the best approach.<br class="">
</blockquote>
<br class="">
Partially agreed, because for the _x Versions in the derived datatype chapter,<br class="">
I would always use MPI_Count and not MPI_Offset, because one never know<br class="">
for which other purpose derived datatypes may be used in the future..<br class="">
The concept is not restricted to Memory and I/O.<br class="">
I/O is only an example why we need to provide these _x Versions.<br class="">
Therefore I prefere:<br class="">
<br class="">
4. leave it. It is the starting point for the other _X versions.<br class="">
<br class="">
Small diff between these two _x routines and the new ones is<br class="">
that for each new one, it does not have a visible Fortran _X, <br class="">
only the overloaded one,<br class="">
and each existing one must get the additional overloaded interface.<br class="">
As far as I understand Fortran, there will be then only one backend routine <br class="">
with two different interfaces, because the backend is the same _X<br class="">
routine for the explicit and for the overloaded MPI_Count routine.<br class="">
<br class="">
Best regards<br class="">
Rolf<br class="">
<br class="">
<br class="">
----- Original Message -----<br class="">
<blockquote type="cite" class="">From: "mpiwg-large-counts" <<a href="mailto:mpiwg-large-counts@lists.mpi-forum.org" class="">mpiwg-large-counts@lists.mpi-forum.org</a>><br class="">
To: "HOLMES Daniel" <<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a>><br class="">
Cc: "Rolf Rabenseifner" <<a href="mailto:rabenseifner@hlrs.de" class="">rabenseifner@hlrs.de</a>>, "mpiwg-large-counts" <<a href="mailto:mpiwg-large-counts@lists.mpi-forum.org" class="">mpiwg-large-counts@lists.mpi-forum.org</a>><br class="">
Sent: Tuesday, November 5, 2019 6:10:25 PM<br class="">
Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements<br class="">
</blockquote>
<br class="">
<blockquote type="cite" class="">Hi Dan,<br class="">
<br class="">
our mailers do not want to work together.<br class="">
I cannot find your "lengthy comment on that issue" in this email.<br class="">
<br class="">
As you can see below, it is flattened.<br class="">
<br class="">
Originally , it was with huge indents, but without any ">".<br class="">
Text looked then líke<br class="">
<br class="">
d<br class="">
d<br class="">
r<br class="">
e<br class="">
s<br class="">
s<br class="">
e<br class="">
s<br class="">
<br class="">
<br class="">
In the telcon, we completely aggreed that the large count versions MPI...._l<br class="">
should keep all MPI_Aint and should add MPI_Aint where it was missing<br class="">
by being wrong (MPI_Alltoall_w and MPI_(Un)pack.<br class="">
<br class="">
We completely agreed that all the MPI_Aint discussion is done.<br class="">
MPI_Aint is an signed integer mis-used to also store absoulte<br class="">
adresses whatever there bits may mean. And we do not change this<br class="">
nor we introduce any new such strange type.<br class="">
<br class="">
And we agreed that there should be for the few derived datatype routines<br class="">
with MPI_Aint an additional set with MPI_Count (or MPI_Offset, which would<br class="">
be wierd) instead of MPI_Aint.<br class="">
These MPI_Count versions do not allow the use of absolute addresses.<br class="">
These MPI_Count versions are needed for the case that sizeof(MPI_Aint)<br class="">
is less than sizeof(MPI_Count), may de 4:8 or 8.12 or 8:16,<br class="">
depending on the memory size per MPI process and the filesystem size.<br class="">
<br class="">
I only proposed in my latest email, that exactly two of them already exist:<br class="">
<br class="">
MPI_Type_get_extent_x and MPI_Type_get_true_extent_x<br class="">
<br class="">
and we should keep them and take the postfix _x for this set of routines<br class="">
instead of throwing them away and reinvernting them again.<br class="">
<br class="">
<br class="">
For ... I would say,<br class="">
- MPI_TYPE_SIZE_X<br class="">
the _l Version should have MPI_Aint size because it is a byte size.<br class="">
- MPI_GET_ELEMENTS_X and MPI_STATUS_SET_ELEMENTS_X<br class="">
in my opinion, we should keep _X, and the _l version<br class="">
is then a duplicate of it.<br class="">
<br class="">
Best regards<br class="">
Rolf<br class="">
<br class="">
----- Original Message -----<br class="">
<blockquote type="cite" class="">From: "HOLMES Daniel" <<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a>><br class="">
To: "Rolf Rabenseifner" <<a href="mailto:rabenseifner@hlrs.de" class="">rabenseifner@hlrs.de</a>><br class="">
Cc: "mpiwg-large-counts" <<a href="mailto:mpiwg-large-counts@lists.mpi-forum.org" class="">mpiwg-large-counts@lists.mpi-forum.org</a>><br class="">
Sent: Tuesday, November 5, 2019 3:18:13 PM<br class="">
Subject: Re: [Mpiwg-large-counts] Large Count - the principles for counts,<br class="">
sizes, and byte and nonbyte displacements<br class="">
</blockquote>
<br class="">
<blockquote type="cite" class="">Hi Rolf (et al),<br class="">
<br class="">
I wrote a lengthy comment on that issue to capture my current understanding of<br class="">
your “really wrong” assertion.<br class="">
<br class="">
Broadly, we agree - I just wanted to write down the reasoning and nuances of<br class="">
that outcome.<br class="">
<br class="">
Cheers,<br class="">
Dan.<br class="">
—<br class="">
Dr Daniel Holmes PhD<br class="">
Architect (HPC Research)<br class="">
<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">d.holmes@epcc.ed.ac.uk</a><<a href="mailto:d.holmes@epcc.ed.ac.uk" class="">mailto:d.holmes@epcc.ed.ac.uk</a>><br class="">
Phone: +44 (0) 131 651 3465<br class="">
Mobile: +44 (0) 7940 524 088<br class="">
Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT<br class="">
—<br class="">
The University of Edinburgh is a charitable body, registered in Scotland, with<br class="">
registration number SC005336.<br class="">
—<br class="">
<br class="">
On 2 Nov 2019, at 09:13, Rolf Rabenseifner via mpiwg-large-counts<br class="">
<<a href="mailto:mpiwg-large-counts@lists.mpi-forum.org" class="">mpiwg-large-counts@lists.mpi-forum.org</a><<a href="mailto:mpiwg-large-counts@lists.mpi-forum.org" class="">mailto:mpiwg-large-counts@lists.mpi-forum.org</a>>><br class="">
wrote:<br class="">
<br class="">
After the Telcon it seems that this ticket is really wrong.<br class="">
Some or all of the routines may/should be kept.<br class="">
And it these routines arean essential part of the future large count concept.<br class="">
<br class="">
Thank you very much for pointing us to this ticket.<br class="">
Rolf<br class="">
<br class="">
----- Jeff Hammond <<a href="mailto:jeff.science@gmail.com" class="">jeff.science@gmail.com</a><<a href="mailto:jeff.science@gmail.com" class="">mailto:jeff.science@gmail.com</a>>><br class="">
wrote:<br class="">
Rolf:<br class="">
<br class="">
Have you looked at<br class="">
<a href="https://github.com/mpiwg-large-count/large-count-issues/issues/6?" class="">https://github.com/mpiwg-large-count/large-count-issues/issues/6?</a><br class="">
<br class="">
Jeff<br class="">
<br class="">
On Fri, Nov 1, 2019 at 1:00 AM Rolf Rabenseifner <rabenseifner@hlrs.de><br class="">
wrote:<br class="">
<br class="">
A small comment on the result of our telcon:<br class="">
- Postfix _l for int -> MPI_Count<br class="">
- Postfix _x for additionally<br class="">
MPI_Aint -> MPI_Count<br class="">
I.e., the additional routines in the derived datatype chapter.<br class="">
<br class="">
Two of them already exist<br class="">
MPI_Type_get_(true)extent_x<br class="">
<br class="">
In Fortran we will have then for the<br class="">
same routine two aliases:<br class="">
<br class="">
- the overload one without _x<br class="">
- and the explicit one with _x<br class="">
<br class="">
For both ones, the internal function name is the same, with _x.<br class="">
<br class="">
Best regards<br class="">
Rolf<br class="">
<br class="">
<br class="">
----- Jeff Hammond <jeff.science@gmail.com> wrote:<br class="">
On Thu, Oct 31, 2019 at 7:48 AM Rolf Rabenseifner <rabenseifner@hlrs.de><br class="">
wrote:<br class="">
<br class="">
Dear all,<br class="">
<br class="">
here my summary as input for our telcon today.<br class="">
<br class="">
In principle, it is a very simple question:<br class="">
<br class="">
with large Counts, do we<br class="">
- keep all MPI_Aint<br class="">
- or do we substitute MPI_Aint by MPI_Count?<br class="">
<br class="">
<br class="">
I haven't been involved as much lately but did we not use MPI_Count for<br class="">
count and element displacements in the large count proposal? We need to<br class="">
use MPI_Aint for offsets into memory because that is what this type is<br class="">
for.<br class="">
<br class="">
Jeff<br class="">
<br class="">
<br class="">
<br class="">
In principle, the MPI Forum answered this question already<br class="">
for MPI-3.0 in 2012 with a clear YES:<br class="">
<br class="">
int MPI_Type_get_extent(MPI_Datatype datatype,<br class="">
MPI_Aint *lb, MPI_Aint *extent)<br class="">
int MPI_Type_get_extent_x(MPI_Datatype datatype,<br class="">
MPI_Count *lb, MPI_Count *extent)<br class="">
<br class="">
About Jeff H. question:<br class="">
If we limit the API to not support MPI_Count<br class="">
means that an MPI implementation has not really such quality options<br class="">
when using I/O fileviews, because the API is restricted to<br class="">
MPI_Aint (which should be implemented based on the, e.g.,<br class="">
64bit memory system).<br class="">
<br class="">
About Jim's comment:<br class="">
<br class="">
Apologies, it's been a while since I looked at the I/O interfaces.<br class="">
If<br class="">
I/O<br class="">
only needs relative displacements that have normal integer<br class="">
semantics,<br class="">
then<br class="">
I don't see why MPI_Count would not work for this purpose. If you<br class="">
have<br class="">
an<br class="">
MPI_Aint that contains a relative displacement, it also has normal<br class="">
integer<br class="">
semantics and can be converted to an MPI_Count.<br class="">
<br class="">
Yes, but this automatically implies that the datatypes must also<br class="">
be able to handle MPI_Count.<br class="">
<br class="">
The only case we really<br class="">
need to look out for is when an integer type contains an absolute<br class="">
address.<br class="">
In those cases, the quantity in the variable cannot be treated as a<br class="">
normal<br class="">
integer and we need special routines to work with it.<br class="">
<br class="">
Yes, this happens when we extend MPI_Aint in the derived datatype<br class="">
routines<br class="">
to MPI_Count.<br class="">
<br class="">
But in principle, this is not a big Problem, as you all could see in<br class="">
the previous emails:<br class="">
<br class="">
- We must do for MPI_Count the same as we did for MPI_Aint,<br class="">
i.e., we'll have long versions of the routines<br class="">
MPI_Get_address, MPI_Aint_diff, MPI_Aint_add<br class="">
<br class="">
- And we must ensure that the type cast from MPI_Aint to<br class="">
MPI_Count works, which is a small new advice to implementors<br class="">
for MPI_Det_address.<br class="">
<br class="">
Therefore again my 4 questions:<br class="">
<br class="">
- Should the new large count routines be prepared for<br class="">
more than 10 or 20 Exabyte files where we need 64/65 or<br class="">
or 65/66 unsigned/signed integers for relative byte<br class="">
displacements or byte counts?<br class="">
If yes, then all MPI_Aint arguments must be substituted by MPI_Count.<br class="">
<br class="">
(In other words, do we want to be prepared for another 25 years of<br class="">
MPI?<br class="">
:-)<br class="">
<br class="">
As stated above, the MPI-Forum already decided 2012 with a YES.<br class="">
<br class="">
- Should we allow that these new routines are also used for memory<br class="">
description,<br class="">
where we typically need only the large MPI_Count "count" arguments?<br class="">
(or should we provide two different new routines for each routine<br class="">
that<br class="">
currently has int Count/... and MPI_Aint disp/... arguments)<br class="">
<br class="">
I expect, that nobody wants to have two different large versions of<br class="">
for example MPI_Type_create_struct.<br class="">
<br class="">
- Should we allow a mix of old and new routines, especially for<br class="">
memory-based<br class="">
usage, that old-style MPI_Get_address is used to retrieve an absolute<br class="">
address and then, e.g., new style MPI_Type_create_struct with<br class="">
MPI_Count blocklength and displacements is used?<br class="">
<br class="">
I expect that forbidding such a mix would be a problem for Software<br class="">
development.<br class="">
Often old-style modules must work together with new-style modules.<br class="">
<br class="">
- Do we want to require for this type cast of MPI_Aint addr into<br class="">
MPI_Count<br class="">
that it is allowed to do this cast with a normal assignment, rather<br class="">
than<br class="">
a special MPI function?<br class="">
<br class="">
I expect yes, because for must usage of MPI_Aint and MPI_Count,<br class="">
it is for relative displacements or byte counts, i.e. for normal<br class="">
integers and therefore automatic type cast between MPI_Aint<br class="">
and MPI_Count is a must.<br class="">
<br class="">
With yes to all four questions, the proposed solution above is<br class="">
the easiest way.<br class="">
<br class="">
Hope to see/hear you today in our telcon.<br class="">
<br class="">
Best regards<br class="">
Rolf<br class="">
</blockquote>
</blockquote>
<br class="">
-- <br class="">
Dr. Rolf Rabenseifner . . . . . . . . . .. <a href="mailto:rabenseifner@hlrs.de" class="">
email rabenseifner@hlrs.de</a> .<br class="">
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 .<br class="">
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 .<br class="">
Head of Dpmt Parallel Computing . . . <a href="http://www.hlrs.de/people/rabenseifner" class="">
www.hlrs.de/people/rabenseifner</a> .<br class="">
Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>