<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Joseph,<div class=""><br class=""></div><div class="">Firstly, I could argue that the implementation problem is caused by an implementation choice: too-lazy initialisation. The MPI Standard requires that an answer be given at a certain point in the execution of the program, so the implementation must do whatever work is needed to provide the answer at the time it is needed. The initialisation work could be lazily delayed until the user actually gets the INFO object associated with the communicator (if they ever do). I don’t particularly like that argument because I don’t like excluding this implementation route without deliberate intent to do so (I assert that no such deliberate intent exists).</div><div class=""><br class=""></div><div class="">Secondly, I could argue that the implementation should simply document its handling of INFO keys in this scenario: it is implementation-specific behaviour. There are lots of ways to phrase that documentation. a) For example, an INFO key will not be ignored if any module reports during communicator creation that it understands that key, even if that module is never actually selected or used. b) For example, when non-standard INFO keys are supplied to a communicator creation procedure, module selection for all operation types will occur during communicator creation instead of being delayed until yada yada ya. c) For example, calling MPI_COMM_GET_INFO will cause module selection to occur if non-standard INFO keys were associated with the communicator during communicator creation or subsequently via MPI_COMM_SET_INFO.</div><div class=""><br class=""></div><div class="">Thirdly, I would quite like the answer to be: INFO keys that only apply to some instances of MPI operations (that covers, but is not identical to, “are understood/used by some modules”) should be applied to the MPI operation rather than to the communicator. This is possible with a persistent scatter operation (for example) but, through an oversight in the MPI Standard, it is not possible for a blocking or nonblocking scatter operation. Thus, unfortunately, this answer is somewhat flawed. Despite that flaw, implementation-specific documentation could encourage the user to replace blocking and nonblocking collective operations with persistent operations and to supply the INFO keys to the persistent collective initialization procedure(s) instead of applying them to the whole communicator. Similar advice could encourage a switch from point-to-point operations to partitioned communication with one partition.<br class=""><div class="">
<meta charset="UTF-8" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br class="Apple-interchange-newline">Cheers,</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Dan.</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">—</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">Dr Daniel Holmes PhD</div>Executive Director<br class="">Chief Technology Officer<br class=""><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">CHI Ltd</div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><a href="mailto:danholmes@chi.scot" class="">danholmes@chi.scot</a></div><div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class=""></div></div><br class="Apple-interchange-newline">
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On 20 Oct 2021, at 20:24, Joseph Schuchart via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">I am working on the fix for Open MPI. The modular infrastructure and lazy initialization there makes this challenging in some places. For example, during communicator creation we don't yet know which collective module will serve which collective operation. Assume we have a hypothetical custom info key "ompi_scatter_blocksize" that is understood by a subset of possible implementations of scatter that the library can choose from at runtime. That choice might not be made until the user actually calls MPI_Scatter. We can track which info keys are supported by the different OMPI modules but we don't know whether the implementation understanding this key will be chosen at the time the user calls MPI_Comm_get_info.<br class=""><br class="">The question that came up: what is the meaning of "ignored" in this context? Is a key ignored if no part of the implementation cares about it or is it "ignored" if some part understands it but the key may not impact the execution of an operation in a particular instance? Obviously, we lean towards the first because otherwise we'd be forced to eagerly set up everything during communicator creation just to get the info keys right... In any case, we can definitely drop unknown keys like "foo" in Ken's example.<br class=""><br class="">Thanks<br class="">Joseph<br class=""><br class="">On 10/19/21 9:44 AM, Raffenetti, Ken via mpi-forum wrote:<br class=""><blockquote type="cite" class="">Thanks Martin. This is my recollection of the discussion, too. I was just surprised to see this behavior in both MPICH and Open MPI. We did come across this sentence in the text:<br class=""><br class=""> An implementation must support info objects as caches for arbitrary (key,value) pairs, regardless of whether it recognizes the key.<br class=""><br class="">But I think we can reasonably understand that to mean an info object that the user creates. I.e. *not* the info object associated with an MPI {comm,win,file} handle. MPICH will fix the MPI_FILE_GET_INFO behavior to do what is stated below.<br class=""><br class="">Thanks,<br class="">Ken<br class=""><br class="">On 10/18/21, 3:26 PM, "Martin Schulz" <<a href="mailto:schulzm@in.tum.de" class="">schulzm@in.tum.de</a>> wrote:<br class=""><br class=""> Hi Ken,<br class=""><br class=""> To my understanding, this is actually a bug in the MPI library or an incorrect implementation according to the standard. We had a very lengthy discussion about this during MPI 4.0 - here is a link to the issue:<br class=""><br class=""> <a href="https://github.com/mpi-forum/mpi-issues/issues/63" class="">https://github.com/mpi-forum/mpi-issues/issues/63</a><br class=""><br class=""> We had in there at some point the following principles that guided the new text:<br class=""><br class=""> Principles:<br class=""> a) the key/value pairs returned by MPI_GET_INFO must always have been supplied by the user and not ignored by the MPI library<br class=""> b) the semantics of supplying an INFO during object creation and during MPI_SET_INFO should be identically defined<br class=""> c) the MPI is permitted, but not required, to 'ignore' any INFO key/value pair supplied by the user<br class=""> d) if an INFO key/value pair is ignored by MPI, then it will not appear in subsequent MPI_GET_INFO calls<br class=""> e) if an INFO key/value pair was not ignored by MPI, then it must appear in subsequent MPI_GET_INFO calls<br class=""><br class=""> (the same was then applied to all get infos)<br class=""><br class=""> d) seems to be the key aspect here. From what I remember, this was a deliberate choice, as the assumption back then was that the user wanted to know what the library actually did with the info (and a library was allowed to not store other values that are not used). Also, we allowed the return of a different value - if a library knows and uses the key "foo", but only uses the first character, the value returned would be "b" - again with the idea that the user wants to know what is used.<br class=""><br class=""> We had actually discussed a second version of MPI_Get_info that would return the actual user supplied values, but we never pursued that - perhaps it is worth reconsidering this again.<br class=""><br class=""> Martin<br class=""><br class=""><br class=""> --<br class=""> Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems<br class=""> Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching<br class=""> Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ)<br class=""> <a href="mailto:schulzm@in.tum.de" class="">Email: schulzm@in.tum.de</a><br class=""><br class=""><br class=""><br class=""> On 18.10.21, 18:12, "mpi-forum on behalf of Raffenetti, Ken via mpi-forum" <<a href="mailto:mpi-forum-bounces@lists.mpi-forum.org" class="">mpi-forum-bounces@lists.mpi-forum.org</a> on behalf of <a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a>> wrote:<br class=""><br class=""> Here's an example program that illustrates the current file info behavior with MPICH and Open MPI. Neither library will abort the application in my tests. MPICH does not recognize the "foo" key, and I would guess the same for Open MPI.<br class=""><br class=""> So is this correct behavior according to the standard? Or a bug? There are users who prefer the current behavior, I suspect in part because there is no attribute caching interface for MPI file objects.<br class=""><br class=""> #include <mpi.h><br class=""> #include <stdlib.h><br class=""> #include <stdio.h><br class=""> #include <string.h><br class=""><br class=""> int main(void) {<br class=""> MPI_Init(NULL, NULL);<br class=""><br class=""> MPI_Info info;<br class=""> MPI_Info_create(&info);<br class=""> MPI_Info_set(info, "foo", "bar");<br class=""><br class=""> MPI_File fh;<br class=""> MPI_File_open(MPI_COMM_SELF, "testfile", MPI_MODE_RDWR | MPI_MODE_CREATE, info, &fh);<br class=""><br class=""> MPI_Info info_used;<br class=""> MPI_File_get_info(fh, &info_used);<br class=""><br class=""> char value[4] = "";<br class=""> int flag;<br class=""> MPI_Info_get(info_used, "foo", 3, value, &flag);<br class=""><br class=""> if (!flag) {<br class=""> printf("key not found\n");<br class=""> MPI_Abort(MPI_COMM_WORLD, 1);<br class=""> }<br class=""><br class=""> if (strcmp(value, "bar")) {<br class=""> printf("value does not match\n");<br class=""> MPI_Abort(MPI_COMM_WORLD, 1);<br class=""> }<br class=""><br class=""> MPI_File_close(&fh);<br class=""> MPI_Info_free(&info);<br class=""> MPI_Info_free(&info_used);<br class=""><br class=""> MPI_Finalize();<br class=""> return 0;<br class=""> }<br class=""><br class=""> On 10/15/21, 2:47 PM, "mpi-forum on behalf of Raffenetti, Ken via mpi-forum" <<a href="mailto:mpi-forum-bounces@lists.mpi-forum.org" class="">mpi-forum-bounces@lists.mpi-forum.org</a> on behalf of <a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a>> wrote:<br class=""><br class=""> We brought this to the forum because during our internal discussion is was shown that ROMIO will return any "ignored" key/value pairs set by the user in MPI_FILE_GET_INFO. Essentially, ROMIO duplicates the users info object, and only adjusts settings for keys it recognizes. Perhaps this is wrong behavior according to MPI-4.0?<br class=""><br class=""> This behavior was introduced into ROMIO somewhat recently in May 2020. There was a user request for it based on how OMPI-IO behaved. <a href="https://github.com/pmodels/mpich/pull/3954" class="">https://github.com/pmodels/mpich/pull/3954</a><br class=""><br class=""> Ken<br class=""><br class=""> On 10/13/21, 3:46 PM, "mpi-forum on behalf of Jim Dinan via mpi-forum" <<a href="mailto:mpi-forum-bounces@lists.mpi-forum.org" class="">mpi-forum-bounces@lists.mpi-forum.org</a> on behalf of <a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a>> wrote:<br class=""><br class=""> Hi Rob,<br class=""> This seems reasonable to me. I think you can draw an analogy to an application setting "mpi_assert_no_any_tag = true" and checking the value returned by MPI_Comm_get_info to decide whether to use MPI_ANY_TAG on the communicator. I would suggest info values like "true" and "false" so that ROMIO can return back the same value passed by the user if the info key is accepted.<br class=""><br class=""> As of MPI 4.0, MPI_File_get_info must return "all hints that are supported by the implementation and have default values specified; any user-supplied hints that were not ignored by the implementation; and any additional hints that were set by the implementation." So, true/false would also allow ROMIO to return a default value for the info key if the user hasn't set it.<br class=""><br class=""> One thing you can't do with info today is fail the operation because of a key/value pair in the info argument. If there is an issue with a particular key/value pair, it is supposed to be ignored.<br class=""><br class=""> ~Jim.<br class=""><br class=""><br class=""> On Mon, Oct 11, 2021 at 2:13 PM Latham, Robert J. via mpi-forum <<a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a>> wrote:<br class=""><br class=""><br class=""> I'd like a way to tell users programmatically about features ROMIO supports.<br class=""><br class=""> One way I imagined doing so was through the info object, though we're having some internal discussion about how kosher that is. Any thoughts from the broader MPI community about an approach like this:<br class=""><br class=""><br class=""> ```<br class=""> MPI_Info_create(&info);<br class=""> MPI_Info_set(info, "romio_feature_xyz", "requested");<br class=""> MPI_File_open(...)<br class=""> MPI_File_get_info(fh, &info_used)<br class=""> MPI_File_get_info(info_used, "romio_feature_xyz", MPI_MAX_INFO_VAL-1,<br class=""> value, &flag);<br class=""><br class=""> ```<br class=""><br class=""> Possible points of contention:<br class=""><br class=""> - we have examples of using hints for clients to pass information to implementations (striping_factor, cb_buffer_size) and examples of implementations passing information to users (cb_nodes on Blue Gene), but we don't have too many examples of two-way hints.<br class=""><br class=""> - will MPI_File_get_info return all info keys or just the ones the implementation understands? The standard dictates what is required to be returned. ROMIO (so de facto standard) will return the union of its internal keys and any keys (known or unknown) passed in by the user.<br class=""><br class=""> - The use of "requested" as a hint value that the implementation can then either set to "enabled" or "disabled" guards the situation where a user code is passing these infos to an older MPI IO implementation. If the hint comes back exactly as given, then the implementation ignored this feature request and it is not supported. If the hint comes back "enabled" then the caller knows something about the implementation.<br class=""><br class=""> Thanks<br class=""> ==rob<br class=""><br class=""><br class=""> _______________________________________________<br class=""> mpi-forum mailing list<br class=""> <a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a><br class=""> <a href="https://lists.mpi-forum.org/mailman/listinfo/mpi-forum" class="">https://lists.mpi-forum.org/mailman/listinfo/mpi-forum</a><br class=""><br class=""> _______________________________________________<br class=""> mpi-forum mailing list<br class=""> <a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a><br class=""> <a href="https://lists.mpi-forum.org/mailman/listinfo/mpi-forum" class="">https://lists.mpi-forum.org/mailman/listinfo/mpi-forum</a><br class=""><br class=""> _______________________________________________<br class=""> mpi-forum mailing list<br class=""> <a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a><br class=""> <a href="https://lists.mpi-forum.org/mailman/listinfo/mpi-forum" class="">https://lists.mpi-forum.org/mailman/listinfo/mpi-forum</a><br class=""><br class=""><br class=""><br class="">_______________________________________________<br class="">mpi-forum mailing list<br class=""><a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a><br class="">https://lists.mpi-forum.org/mailman/listinfo/mpi-forum<br class=""></blockquote><br class="">_______________________________________________<br class="">mpi-forum mailing list<br class=""><a href="mailto:mpi-forum@lists.mpi-forum.org" class="">mpi-forum@lists.mpi-forum.org</a><br class="">https://lists.mpi-forum.org/mailman/listinfo/mpi-forum<br class=""></div></div></blockquote></div><br class=""></div></body></html>